Problem description

In this practice we will develop several convolutional neural network (CNNs) models trying to predict an animal given its image.

Dataset description

We will use the "Animals dataset" that contains about 15K pictures of five types of animals: cats, dogs, elephants, horses & lions, collected from Google Images.

Configure the environment

Set the following variables with your specific configuration:

# Set Random Seed to ensure reproducibility of the experiments
RANDOM_SEED = 54

# Directory parameters
DATASET_PATH = './datasets/animals'
OUT_MODELS_FOLDER = "./output"

# Image parameters
# Rescaling
IMG_HEIGHT = 180
IMG_WIDTH = 180

# Model parameters
BATCH_SIZE = 32

Prepare the environment

import os

import keras
from keras import utils, layers, optimizers, applications

import keras_tuner

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import pandas as pd
from tabulate import tabulate

from itertools import product

# AUXILIARY FUNCTIONS
def plot(history):
    accuracy = history.history["accuracy"]
    val_accuracy = history.history["val_accuracy"]
    loss = history.history["loss"]
    val_loss = history.history["val_loss"]

    epochs = range(1, len(accuracy) + 1)

    plt.plot(epochs, accuracy, "b-o", label="Training accuracy")
    plt.plot(epochs, val_accuracy, "r-o", label="Validation accuracy")
    plt.title("Training and validation accuracy")
    plt.legend()
    plt.figure()

    plt.plot(epochs, loss, "b-o", label="Training loss")
    plt.plot(epochs, val_loss, "r-o", label="Validation loss")
    plt.title("Training and validation loss")
    plt.legend()
    plt.show()

def plot_distribution(dataset):
    ax = sns.countplot(dataset.label)
    plt.xticks(ticks=labels, labels = product)
    plt.title('Distribution of Target Classes, Train', fontsize=18, y=1.02)
    # Loops through each bar in the plot and adds an annotation with the height of the bar
    for bar in ax.patches:
        ax.annotate(format(bar.get_height(), '.0f'),(bar.get_x() + bar.get_width() / 2,bar.get_height()), 
                    ha='center', va='center',size=15, xytext=(0, 8),textcoords='offset points')
    plt.show()

def plot_results(model_names, metric_name, validation_results, test_results):
    # Set width of bar
    bar_width = 0.35

    # Set position of bar on X axis
    r1 = np.arange(len(validation_results))
    r2 = [x + bar_width for x in r1]

    # Make the plot
    plt.bar(r1, validation_results, color='b', width=bar_width, edgecolor='grey', label='Validation '+metric_name)
    plt.bar(r2, test_results, color='r', width=bar_width, edgecolor='grey', label='Test '+metric_name)

    # Add xticks on the middle of the group bars
    plt.xlabel('Models', fontweight='bold')
    plt.xticks([r + bar_width/2 for r in range(len(validation_results))], model_names)

    # Add ylabel
    plt.ylabel(metric_name, fontweight='bold')

    # Add title
    plt.title('Validation and Test '+metric_name+' for Different Models')

    # Add legend
    plt.legend()

    # Show plot
    plt.show()

2024-03-05 18:47:31.839478: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-05 18:47:31.839533: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-05 18:47:31.888113: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-05 18:47:31.991727: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-05 18:47:32.967677: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

1. Preprocess the dataset

This step will load and prepare the dataset for training and evaluation.

Some comments about the dataset:

• The dataset is already divided into train and validation subsets. The val part of the dataset will be left as a test set
• The train subset will be further subdivided, leaving 10% of the data for validation.
• Each image is standarized to a more manageable size 180x180x3 tensor (RGB).
• Images are normalized between [0, 1] to facilitate the learning process of the networks.
• Each label consists in a number asociated to a kind of animal (cat, dog, elephant, horse and lion).
• In each batch iteration, 32 (BATCH_SIZE) images are seen.
• Training, validation and test sets will be a (tf.data.Dataset)

# Set the path for the dataset
assert os.path.exists(DATASET_PATH), f'Verify the dataset is located at {DATASET_PATH}'

def normalization(image,label):
    image = tf.cast(image/255. ,tf.float32)
    return image,label

# Load train, validation and test datasets
train_dataset, val_dataset = utils.image_dataset_from_directory(DATASET_PATH+"/train",                             
                                                   image_size=(IMG_HEIGHT, IMG_WIDTH),
                                                   label_mode="categorical",
                                                   batch_size=BATCH_SIZE,
                                                   seed=RANDOM_SEED,
                                                   subset="both", # Return both training and validation sets
                                                   validation_split=0.1)
test_dataset = utils.image_dataset_from_directory(DATASET_PATH+"/val", # val folder will be left as a test set in order to compare the generalization capability of the different models.
                                                  image_size=(IMG_HEIGHT, IMG_WIDTH),
                                                   label_mode="categorical",
                                                   batch_size=BATCH_SIZE,
                                                   seed=RANDOM_SEED)

# Normalization
train_dataset = train_dataset.map(normalization)
val_dataset = val_dataset.map(normalization)
test_dataset = test_dataset.map(normalization)

Found 13474 files belonging to 5 classes.
Using 12127 files for training.
Using 1347 files for validation.

2024-03-05 18:47:51.554880: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.721749: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.721998: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.725028: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.725247: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.725445: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.927157: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.927445: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.927459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2022] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2024-03-05 18:47:51.927669: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:07:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-05 18:47:51.927695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9711 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:07:00.0, compute capability: 8.6

Found 1497 files belonging to 5 classes.

# Visualize sample images
for batch_images, batch_labels in train_dataset.take(5):
    print("Image shape: ", batch_images[0].numpy().shape)
    print("Label: ", batch_labels[0].numpy())
    plt.imshow(batch_images[0].numpy())
    plt.show()

Image shape:  (180, 180, 3)
Label:  [0. 0. 0. 0. 1.]

Image shape:  (180, 180, 3)
Label:  [0. 0. 0. 1. 0.]

Image shape:  (180, 180, 3)
Label:  [0. 1. 0. 0. 0.]

Image shape:  (180, 180, 3)
Label:  [0. 0. 1. 0. 0.]

Image shape:  (180, 180, 3)
Label:  [0. 0. 0. 1. 0.]

2. Develop a custom convolutional model for the classification problem.

The network considered for this lab session is a Convolutional Neural Network (CNN), well-suited models for a wide range of image classification tasks due to multiple factors:

• Local connectivity: the operation of the convolution with filters of a smaller size than the input means that the connectivity is local, which reduces the number of operations by performing computations only in subregions and not having connections to all the input, as in fully connected networks. In addition, it allows each neuron to specialize in a different feature, allowing complex interactions to be modeled as a function of the activations of different neurons.

• Shared weights: all neurons operating on the same feature map employ filters with the same coefficients (same kernel), further reducing the total number of parameters. Without this property, the filter coefficients at each point during the convolution process would vary. Intuitively, in the case of an image, it is understood that, for example, if a filter is useful for detecting edges at a certain position, it is also useful for detecting edges at any other position of the image.

• Translation invariance: shared weights cause the data representations to be moved in the feature map when the input is also moved.

• Spatial hierarchies of patterns: the network learns how to represent the input information from the lowest level up through multiple layers to create increasingly sophisticated representations of the data. For example, the layers closest to the input are responsible for learning what an edge or a corner is, while later layers receive the information from these neurons and combine them to decide that if there are two vertical and horizontal edges in close proximity, parallel two by two, then you are dealing with a rectangle. This feature allows reusing networks that were trained on one dataset to apply them in other domains by changing only the deepest layers, which are specialized in the training domain.

• No need for feature engineering: filters are automatically learnt during the training process, so we do not need to hand-craft features in order to train efficiently the network. In fact, it is the same network the one that decides which features are most important by adapting its filters to the current problem.

The basic architecture for our initial experiments is composed of Blocks of convolution and max-pooling (feature extraction block), followed by dense layers (classification block). A flatten layer is also needed between the feature extraction blocks and the classification block because the latter only accepts a vector as input, not matrices nor tensors.

In order to determine the best architecture of the model (convolutional layers, pooling layers, number of filters, size of kernels, etc.) several experiments are carried out below to find the architecture which best fits to out classification problem. Once the best architecture is obtained, we will check if there is a problem of overfitting and, if so, will apply regularization techniques to check if the performance of the network is affected.

Accuracy will be the metric in use to evaluate the performance of the trained classifiers. As it reflects the overall correctness of the model across all classes, it is particularly suitable since classes in the Animals dataset are balanced, meaning they have roughly the same number of samples (≈2700 instances per class). F1-score, as well as Precision and Recall, are also computed for comparative purposes.

Note that the most relevant trained models will be saved into ./output folder making possible a later analysis.

First, let's see whether CNNs can provide fair results with a single test experiment.

# First hand-crafted experiment
inputs = keras.Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3))

# Feature extraction block
x = layers.Conv2D(filters=32, kernel_size=(3,3), activation="relu")(inputs)
x = layers.Conv2D(filters=32, kernel_size=(3,3), activation="relu")(x)
x = layers.MaxPooling2D(pool_size=2)(x)
x = layers.Dropout(0.5)(x)
x = layers.Conv2D(filters=64, kernel_size=(3,3), activation="relu")(x)
x = layers.Conv2D(filters=64, kernel_size=(3,3), activation="relu")(x)
x = layers.MaxPooling2D(pool_size=2)(x)

# Classification block
x = layers.Flatten()(x)
x = layers.Dense(128, activation="relu")(x)

# Output layer
outputs = layers.Dense(5, activation="softmax")(x)

# Prepare model
model = keras.Model(inputs=inputs, outputs=outputs)

model.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d (Conv2D)             (None, 178, 178, 32)      896       
                                                                 
 conv2d_1 (Conv2D)           (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d (MaxPooling2  (None, 88, 88, 32)        0         
 D)                                                              
                                                                 
 dropout (Dropout)           (None, 88, 88, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 86, 86, 64)        18496     
                                                                 
 conv2d_3 (Conv2D)           (None, 84, 84, 64)        36928     
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 42, 42, 64)        0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 112896)            0         
                                                                 
 dense (Dense)               (None, 128)               14450816  
                                                                 
 dense_1 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 14517029 (55.38 MB)
Trainable params: 14517029 (55.38 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath=OUT_MODELS_FOLDER+"/convnet_from_scratch.keras",
        save_best_only=True,
        monitor="val_loss")
]

model.compile(optimizer="adam",
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()])

history = model.fit(train_dataset, epochs=25, batch_size=BATCH_SIZE,
          validation_data=val_dataset,
          callbacks=callbacks)

Epoch 1/25

2024-02-18 18:18:12.737855: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:961] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
2024-02-18 18:18:13.574700: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-18 18:18:15.424864: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f9ec4017140 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-18 18:18:15.424898: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-18 18:18:15.436865: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1708276695.539743    7485 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 57s 115ms/step - loss: 1.3022 - accuracy: 0.4732 - f1_score: 0.4696 - precision: 0.7294 - recall: 0.2268 - val_loss: 0.8808 - val_accuracy: 0.6696 - val_f1_score: 0.6662 - val_precision: 0.8268 - val_recall: 0.4217
Epoch 2/25
379/379 [==============================] - 37s 96ms/step - loss: 0.7431 - accuracy: 0.7043 - f1_score: 0.7038 - precision: 0.7980 - recall: 0.5762 - val_loss: 0.6680 - val_accuracy: 0.7402 - val_f1_score: 0.7395 - val_precision: 0.8300 - val_recall: 0.5909
Epoch 3/25
379/379 [==============================] - 37s 97ms/step - loss: 0.5367 - accuracy: 0.7878 - f1_score: 0.7877 - precision: 0.8397 - recall: 0.7268 - val_loss: 0.5879 - val_accuracy: 0.7721 - val_f1_score: 0.7735 - val_precision: 0.8054 - val_recall: 0.7342
Epoch 4/25
379/379 [==============================] - 26s 68ms/step - loss: 0.3771 - accuracy: 0.8512 - f1_score: 0.8512 - precision: 0.8831 - recall: 0.8173 - val_loss: 0.6857 - val_accuracy: 0.7595 - val_f1_score: 0.7590 - val_precision: 0.8010 - val_recall: 0.6904
Epoch 5/25
379/379 [==============================] - 24s 63ms/step - loss: 0.2695 - accuracy: 0.8996 - f1_score: 0.8998 - precision: 0.9173 - recall: 0.8813 - val_loss: 0.9452 - val_accuracy: 0.7379 - val_f1_score: 0.7379 - val_precision: 0.7634 - val_recall: 0.7164
Epoch 6/25
379/379 [==============================] - 24s 63ms/step - loss: 0.1947 - accuracy: 0.9264 - f1_score: 0.9265 - precision: 0.9377 - recall: 0.9176 - val_loss: 0.9991 - val_accuracy: 0.7335 - val_f1_score: 0.7376 - val_precision: 0.7488 - val_recall: 0.7149
Epoch 7/25
379/379 [==============================] - 24s 62ms/step - loss: 0.1387 - accuracy: 0.9531 - f1_score: 0.9531 - precision: 0.9589 - recall: 0.9475 - val_loss: 1.0871 - val_accuracy: 0.7632 - val_f1_score: 0.7650 - val_precision: 0.7735 - val_recall: 0.7580
Epoch 8/25
379/379 [==============================] - 25s 63ms/step - loss: 0.1026 - accuracy: 0.9642 - f1_score: 0.9642 - precision: 0.9677 - recall: 0.9603 - val_loss: 1.3007 - val_accuracy: 0.7691 - val_f1_score: 0.7683 - val_precision: 0.7735 - val_recall: 0.7632
Epoch 9/25
379/379 [==============================] - 24s 63ms/step - loss: 0.0989 - accuracy: 0.9700 - f1_score: 0.9700 - precision: 0.9734 - recall: 0.9670 - val_loss: 1.3157 - val_accuracy: 0.7684 - val_f1_score: 0.7678 - val_precision: 0.7752 - val_recall: 0.7654
Epoch 10/25
379/379 [==============================] - 25s 63ms/step - loss: 0.0611 - accuracy: 0.9806 - f1_score: 0.9806 - precision: 0.9815 - recall: 0.9798 - val_loss: 1.4863 - val_accuracy: 0.7610 - val_f1_score: 0.7611 - val_precision: 0.7647 - val_recall: 0.7550
Epoch 11/25
379/379 [==============================] - 26s 67ms/step - loss: 0.0651 - accuracy: 0.9786 - f1_score: 0.9786 - precision: 0.9804 - recall: 0.9768 - val_loss: 1.6051 - val_accuracy: 0.7520 - val_f1_score: 0.7494 - val_precision: 0.7608 - val_recall: 0.7461
Epoch 12/25
379/379 [==============================] - 24s 63ms/step - loss: 0.0531 - accuracy: 0.9833 - f1_score: 0.9834 - precision: 0.9846 - recall: 0.9824 - val_loss: 1.6453 - val_accuracy: 0.7743 - val_f1_score: 0.7770 - val_precision: 0.7755 - val_recall: 0.7721
Epoch 13/25
379/379 [==============================] - 24s 63ms/step - loss: 0.0525 - accuracy: 0.9847 - f1_score: 0.9847 - precision: 0.9864 - recall: 0.9838 - val_loss: 1.4775 - val_accuracy: 0.7550 - val_f1_score: 0.7559 - val_precision: 0.7590 - val_recall: 0.7506
Epoch 14/25
379/379 [==============================] - 24s 63ms/step - loss: 0.0309 - accuracy: 0.9905 - f1_score: 0.9905 - precision: 0.9913 - recall: 0.9899 - val_loss: 1.6102 - val_accuracy: 0.7661 - val_f1_score: 0.7672 - val_precision: 0.7655 - val_recall: 0.7535
Epoch 15/25
379/379 [==============================] - 24s 63ms/step - loss: 0.0370 - accuracy: 0.9884 - f1_score: 0.9884 - precision: 0.9891 - recall: 0.9878 - val_loss: 2.2623 - val_accuracy: 0.7283 - val_f1_score: 0.7342 - val_precision: 0.7324 - val_recall: 0.7253
Epoch 16/25
379/379 [==============================] - 25s 66ms/step - loss: 0.0406 - accuracy: 0.9882 - f1_score: 0.9882 - precision: 0.9887 - recall: 0.9877 - val_loss: 1.5822 - val_accuracy: 0.7780 - val_f1_score: 0.7796 - val_precision: 0.7807 - val_recall: 0.7743
Epoch 17/25
379/379 [==============================] - 32s 82ms/step - loss: 0.0388 - accuracy: 0.9877 - f1_score: 0.9877 - precision: 0.9888 - recall: 0.9870 - val_loss: 1.7484 - val_accuracy: 0.7654 - val_f1_score: 0.7685 - val_precision: 0.7699 - val_recall: 0.7602
Epoch 18/25
379/379 [==============================] - 34s 88ms/step - loss: 0.0380 - accuracy: 0.9884 - f1_score: 0.9884 - precision: 0.9887 - recall: 0.9876 - val_loss: 1.6411 - val_accuracy: 0.7803 - val_f1_score: 0.7821 - val_precision: 0.7849 - val_recall: 0.7773
Epoch 19/25
379/379 [==============================] - 32s 83ms/step - loss: 0.0340 - accuracy: 0.9894 - f1_score: 0.9894 - precision: 0.9897 - recall: 0.9890 - val_loss: 1.8087 - val_accuracy: 0.7580 - val_f1_score: 0.7613 - val_precision: 0.7616 - val_recall: 0.7565
Epoch 20/25
379/379 [==============================] - 34s 87ms/step - loss: 0.0382 - accuracy: 0.9890 - f1_score: 0.9890 - precision: 0.9894 - recall: 0.9886 - val_loss: 1.5057 - val_accuracy: 0.7684 - val_f1_score: 0.7675 - val_precision: 0.7744 - val_recall: 0.7669
Epoch 21/25
379/379 [==============================] - 34s 88ms/step - loss: 0.0280 - accuracy: 0.9915 - f1_score: 0.9915 - precision: 0.9919 - recall: 0.9914 - val_loss: 1.7552 - val_accuracy: 0.7654 - val_f1_score: 0.7674 - val_precision: 0.7691 - val_recall: 0.7617
Epoch 22/25
379/379 [==============================] - 32s 83ms/step - loss: 0.0279 - accuracy: 0.9913 - f1_score: 0.9913 - precision: 0.9915 - recall: 0.9908 - val_loss: 1.7076 - val_accuracy: 0.7676 - val_f1_score: 0.7669 - val_precision: 0.7708 - val_recall: 0.7639
Epoch 23/25
379/379 [==============================] - 34s 88ms/step - loss: 0.0226 - accuracy: 0.9941 - f1_score: 0.9941 - precision: 0.9941 - recall: 0.9938 - val_loss: 1.8806 - val_accuracy: 0.7669 - val_f1_score: 0.7650 - val_precision: 0.7710 - val_recall: 0.7624
Epoch 24/25
379/379 [==============================] - 33s 86ms/step - loss: 0.0306 - accuracy: 0.9918 - f1_score: 0.9917 - precision: 0.9920 - recall: 0.9913 - val_loss: 1.8313 - val_accuracy: 0.7661 - val_f1_score: 0.7660 - val_precision: 0.7704 - val_recall: 0.7624
Epoch 25/25
379/379 [==============================] - 36s 95ms/step - loss: 0.0175 - accuracy: 0.9949 - f1_score: 0.9949 - precision: 0.9953 - recall: 0.9946 - val_loss: 1.9837 - val_accuracy: 0.7647 - val_f1_score: 0.7667 - val_precision: 0.7684 - val_recall: 0.7610

plot(history)

Architecture tuning

The previous results show that this basic architecture could be suitable for this problem, reaching a good test accuracy (0.81). In order to maximize the network's performance, in this section 15 different variations following a Random Search approach are tested using KerasTuner, a general-purpose hyperparameter tuning library. The different parameters tested on the network are detailed below:

• Depth of the first convolutional layer (input_units): For successives convolutional layers, depth is doubled while the resolutions are halved. Parameters tested: [32, 64]
• Kernel size of the convolutional layers (kernel_size). Parameters tested: [3, 5, 7, 9]
• Blocks of convolution and max-pooling (n_layers). Parameters tested: [1, 2, 3]
• Number of convolutional layers followed by a max-pooling layer (n_conv). Parameters tested: [1, 2, 3]
• Number of dense layers in the classification block (n_connections). Parameters tested: [1, 2, 3, 4]
• Number of units in a dense layer (n_nodes). Parameters tested: [128, 256, 512, 1024]

def test_architecture(hp):  # random search passes this hyperparameter() object 
    # Get hyperparameters
    input_filters = hp.Int('input_units',min_value=32, max_value=64, step=32)
    kernel_size = hp.Int('kernel_size',min_value=3, max_value=9, step=2)
    
    # Feature extraction block
    inputs = keras.Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3))
    # Convolution layer
    x = layers.Conv2D(filters=input_filters, kernel_size=kernel_size, activation="relu")(inputs)
    x = layers.MaxPooling2D(pool_size=2)(x)

    # (Adjustable) More convolution layers and convolution per pooling
    for i in range(hp.Int('n_layers', 1, 3)):  # adding variation of layers.
        for j in range(hp.Int('n_conv', 1, 3)):  # adding variation of conv per pooling.
            x = layers.Conv2D(filters=(input_filters*(2**i)), # increase depth by power of 2 for each conv layer
                               kernel_size=kernel_size, activation="relu")(x)
        
        x = layers.MaxPooling2D(pool_size=2)(x)

    # Flatten layer
    x = layers.Flatten()(x)

    ##################################################
    # Classification block
    # Fully connected layers
    for i in range(hp.Int('n_connections', 1, 4)):
        x = layers.Dense(hp.Choice(f'n_nodes',
                                  values=[128, 256, 512, 1024]), activation="relu")(x)
    
    # Output layer
    outputs = layers.Dense(5, activation="softmax")(x)

    # Prepare model
    model = keras.Model(inputs=inputs, outputs=outputs)

    model.compile(optimizer="adam",
        loss="categorical_crossentropy",
        metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
    )

    return model

# Generate tuner
tuner = keras_tuner.RandomSearch(
    hypermodel=test_architecture,
    objective="val_accuracy",
    max_trials=15,
    overwrite=True,
    directory=OUT_MODELS_FOLDER,
    project_name="define_arch",
    seed=RANDOM_SEED
)

tuner.search_space_summary()

Search space summary
Default search space size: 6
input_units (Int)
{'default': None, 'conditions': [], 'min_value': 32, 'max_value': 64, 'step': 32, 'sampling': 'linear'}
kernel_size (Int)
{'default': None, 'conditions': [], 'min_value': 3, 'max_value': 9, 'step': 2, 'sampling': 'linear'}
n_layers (Int)
{'default': None, 'conditions': [], 'min_value': 1, 'max_value': 3, 'step': 1, 'sampling': 'linear'}
n_conv (Int)
{'default': None, 'conditions': [], 'min_value': 1, 'max_value': 3, 'step': 1, 'sampling': 'linear'}
n_connections (Int)
{'default': None, 'conditions': [], 'min_value': 1, 'max_value': 4, 'step': 1, 'sampling': 'linear'}
n_nodes (Choice)
{'default': 128, 'conditions': [], 'values': [128, 256, 512, 1024], 'ordered': True}

# Search the best architecture
tuner.search(train_dataset, epochs=25, validation_data=val_dataset,
        callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', mode="min", patience=2)])

Trial 15 Complete [00h 00m 47s]
val_accuracy: 0.7988121509552002

Best val_accuracy So Far: 0.8381588459014893
Total elapsed time: 00h 09m 45s

# Get the top model
best_model = tuner.get_best_models()[0]
best_model.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d (Conv2D)             (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d (MaxPooling2  (None, 89, 89, 64)        0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 43, 43, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 20, 20, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_3 (Conv2D)           (None, 18, 18, 192)       221376    
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 9, 9, 192)         0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 15552)             0         
                                                                 
 dense (Dense)               (None, 128)               1990784   
                                                                 
 dense_1 (Dense)             (None, 128)               16512     
                                                                 
 dense_2 (Dense)             (None, 128)               16512     
                                                                 
 dense_3 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 2358405 (9.00 MB)
Trainable params: 2358405 (9.00 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

As we can see, the best architecture provides a val_accuracy of 0.84, a really good start. In the following cells we can see a summary of the tuning results.

# Print a summary of the search results.
tuner.results_summary()

Results summary
Results in ./output/define_arch
Showing 10 best trials
Objective(name="val_accuracy", direction="max")

Trial 00 summary
Hyperparameters:
input_units: 64
kernel_size: 3
n_layers: 3
n_conv: 1
n_connections: 3
n_nodes: 128
Score: 0.8381588459014893

Trial 05 summary
Hyperparameters:
input_units: 64
kernel_size: 3
n_layers: 3
n_conv: 1
n_connections: 2
n_nodes: 128
Score: 0.8195990920066833

Trial 07 summary
Hyperparameters:
input_units: 32
kernel_size: 3
n_layers: 2
n_conv: 2
n_connections: 1
n_nodes: 512
Score: 0.8129175901412964

Trial 14 summary
Hyperparameters:
input_units: 32
kernel_size: 3
n_layers: 3
n_conv: 2
n_connections: 3
n_nodes: 1024
Score: 0.7988121509552002

Trial 11 summary
Hyperparameters:
input_units: 64
kernel_size: 3
n_layers: 2
n_conv: 1
n_connections: 4
n_nodes: 1024
Score: 0.7869338989257812

Trial 12 summary
Hyperparameters:
input_units: 64
kernel_size: 3
n_layers: 3
n_conv: 2
n_connections: 3
n_nodes: 256
Score: 0.7824795842170715

Trial 04 summary
Hyperparameters:
input_units: 64
kernel_size: 3
n_layers: 1
n_conv: 2
n_connections: 1
n_nodes: 256
Score: 0.7802523970603943

Trial 13 summary
Hyperparameters:
input_units: 32
kernel_size: 5
n_layers: 2
n_conv: 3
n_connections: 3
n_nodes: 128
Score: 0.7468448281288147

Trial 01 summary
Hyperparameters:
input_units: 32
kernel_size: 5
n_layers: 3
n_conv: 2
n_connections: 2
n_nodes: 256
Score: 0.20044542849063873

Trial 02 summary
Hyperparameters:
input_units: 32
kernel_size: 3
n_layers: 3
n_conv: 3
n_connections: 4
n_nodes: 256
Score: 0.20044542849063873

# Show best architecture found
best_hps = tuner.get_best_hyperparameters()[0]
print(best_hps.values)

{'input_units': 64, 'kernel_size': 3, 'n_layers': 3, 'n_conv': 1, 'n_connections': 3, 'n_nodes': 128}

ARCHITECTURE 1

Train the best model

Using the previous architecture:

• Depth of the first convolutional layer (input_units): 64
• Kernel size of the convolutional layers (kernel_size): 3
• Blocks of convolution and max-pooling (n_layers): 3
• Number of convolutional layers followed by a max-pooling layer (n_conv): 1
• Number of dense layers in the classification block (n_connections): 3
• Number of units in a dense layer (n_nodes): 128

we will perform a full training without EarlyStopping to more clearly see the effects of possible overfitting and whether our model needs regularization.

# Build the model with the best hyperparameters.
model = test_architecture(best_hps)

callbacks = [
    keras.callbacks.ModelCheckpoint(
        filepath=OUT_MODELS_FOLDER+"/cnn_best_architecture.keras",
        save_best_only=True,
        monitor="val_loss")
]

history = model.fit(
    train_dataset,
    epochs=25,
    validation_data=val_dataset,
    callbacks=callbacks)

Epoch 1/25
379/379 [==============================] - 8s 17ms/step - loss: 1.0257 - accuracy: 0.5602 - f1_score: 0.5615 - precision_1: 0.7519 - recall_1: 0.3681 - val_loss: 0.6426 - val_accuracy: 0.7342 - val_f1_score: 0.7306 - val_precision_1: 0.7894 - val_recall_1: 0.6622
Epoch 2/25
379/379 [==============================] - 7s 17ms/step - loss: 0.5982 - accuracy: 0.7595 - f1_score: 0.7588 - precision_1: 0.8138 - recall_1: 0.6940 - val_loss: 0.5558 - val_accuracy: 0.7803 - val_f1_score: 0.7798 - val_precision_1: 0.8295 - val_recall_1: 0.7298
Epoch 3/25
379/379 [==============================] - 6s 16ms/step - loss: 0.4969 - accuracy: 0.7961 - f1_score: 0.7958 - precision_1: 0.8423 - recall_1: 0.7477 - val_loss: 0.5437 - val_accuracy: 0.7803 - val_f1_score: 0.7803 - val_precision_1: 0.8277 - val_recall_1: 0.7238
Epoch 4/25
379/379 [==============================] - 6s 16ms/step - loss: 0.3903 - accuracy: 0.8423 - f1_score: 0.8425 - precision_1: 0.8746 - recall_1: 0.8089 - val_loss: 0.4859 - val_accuracy: 0.8196 - val_f1_score: 0.8200 - val_precision_1: 0.8477 - val_recall_1: 0.8018
Epoch 5/25
379/379 [==============================] - 7s 17ms/step - loss: 0.3214 - accuracy: 0.8752 - f1_score: 0.8753 - precision_1: 0.8974 - recall_1: 0.8514 - val_loss: 0.4734 - val_accuracy: 0.8330 - val_f1_score: 0.8334 - val_precision_1: 0.8604 - val_recall_1: 0.8099
Epoch 6/25
379/379 [==============================] - 6s 16ms/step - loss: 0.2667 - accuracy: 0.8976 - f1_score: 0.8978 - precision_1: 0.9158 - recall_1: 0.8822 - val_loss: 0.5556 - val_accuracy: 0.8218 - val_f1_score: 0.8212 - val_precision_1: 0.8364 - val_recall_1: 0.8122
Epoch 7/25
379/379 [==============================] - 6s 16ms/step - loss: 0.2252 - accuracy: 0.9141 - f1_score: 0.9143 - precision_1: 0.9264 - recall_1: 0.8985 - val_loss: 0.5276 - val_accuracy: 0.8315 - val_f1_score: 0.8328 - val_precision_1: 0.8427 - val_recall_1: 0.8233
Epoch 8/25
379/379 [==============================] - 6s 16ms/step - loss: 0.1747 - accuracy: 0.9345 - f1_score: 0.9346 - precision_1: 0.9438 - recall_1: 0.9271 - val_loss: 0.5451 - val_accuracy: 0.8241 - val_f1_score: 0.8271 - val_precision_1: 0.8360 - val_recall_1: 0.8174
Epoch 9/25
379/379 [==============================] - 6s 16ms/step - loss: 0.1418 - accuracy: 0.9476 - f1_score: 0.9476 - precision_1: 0.9540 - recall_1: 0.9411 - val_loss: 0.6908 - val_accuracy: 0.8359 - val_f1_score: 0.8375 - val_precision_1: 0.8420 - val_recall_1: 0.8307
Epoch 10/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0971 - accuracy: 0.9644 - f1_score: 0.9644 - precision_1: 0.9681 - recall_1: 0.9624 - val_loss: 0.7012 - val_accuracy: 0.8456 - val_f1_score: 0.8460 - val_precision_1: 0.8552 - val_recall_1: 0.8419
Epoch 11/25
379/379 [==============================] - 7s 17ms/step - loss: 0.1012 - accuracy: 0.9640 - f1_score: 0.9640 - precision_1: 0.9671 - recall_1: 0.9611 - val_loss: 0.8722 - val_accuracy: 0.8218 - val_f1_score: 0.8228 - val_precision_1: 0.8257 - val_recall_1: 0.8196
Epoch 12/25
379/379 [==============================] - 7s 17ms/step - loss: 0.0809 - accuracy: 0.9715 - f1_score: 0.9715 - precision_1: 0.9735 - recall_1: 0.9694 - val_loss: 0.8575 - val_accuracy: 0.8270 - val_f1_score: 0.8294 - val_precision_1: 0.8285 - val_recall_1: 0.8248
Epoch 13/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0738 - accuracy: 0.9761 - f1_score: 0.9761 - precision_1: 0.9777 - recall_1: 0.9744 - val_loss: 0.7602 - val_accuracy: 0.8396 - val_f1_score: 0.8411 - val_precision_1: 0.8432 - val_recall_1: 0.8382
Epoch 14/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0620 - accuracy: 0.9793 - f1_score: 0.9793 - precision_1: 0.9803 - recall_1: 0.9786 - val_loss: 0.7618 - val_accuracy: 0.8099 - val_f1_score: 0.8102 - val_precision_1: 0.8171 - val_recall_1: 0.8025
Epoch 15/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0445 - accuracy: 0.9851 - f1_score: 0.9851 - precision_1: 0.9860 - recall_1: 0.9842 - val_loss: 0.8501 - val_accuracy: 0.8241 - val_f1_score: 0.8238 - val_precision_1: 0.8307 - val_recall_1: 0.8196
Epoch 16/25
379/379 [==============================] - 7s 17ms/step - loss: 0.0593 - accuracy: 0.9805 - f1_score: 0.9806 - precision_1: 0.9817 - recall_1: 0.9800 - val_loss: 0.7722 - val_accuracy: 0.8471 - val_f1_score: 0.8485 - val_precision_1: 0.8510 - val_recall_1: 0.8441
Epoch 17/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0409 - accuracy: 0.9866 - f1_score: 0.9866 - precision_1: 0.9872 - recall_1: 0.9859 - val_loss: 0.8037 - val_accuracy: 0.8419 - val_f1_score: 0.8426 - val_precision_1: 0.8447 - val_recall_1: 0.8396
Epoch 18/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0354 - accuracy: 0.9886 - f1_score: 0.9886 - precision_1: 0.9891 - recall_1: 0.9878 - val_loss: 0.9656 - val_accuracy: 0.8151 - val_f1_score: 0.8133 - val_precision_1: 0.8211 - val_recall_1: 0.8144
Epoch 19/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0363 - accuracy: 0.9888 - f1_score: 0.9888 - precision_1: 0.9893 - recall_1: 0.9880 - val_loss: 0.9228 - val_accuracy: 0.8241 - val_f1_score: 0.8257 - val_precision_1: 0.8282 - val_recall_1: 0.8233
Epoch 20/25
379/379 [==============================] - 7s 17ms/step - loss: 0.0403 - accuracy: 0.9865 - f1_score: 0.9865 - precision_1: 0.9866 - recall_1: 0.9859 - val_loss: 0.8391 - val_accuracy: 0.8330 - val_f1_score: 0.8337 - val_precision_1: 0.8369 - val_recall_1: 0.8307
Epoch 21/25
379/379 [==============================] - 7s 17ms/step - loss: 0.0274 - accuracy: 0.9918 - f1_score: 0.9919 - precision_1: 0.9922 - recall_1: 0.9912 - val_loss: 1.1159 - val_accuracy: 0.8137 - val_f1_score: 0.8176 - val_precision_1: 0.8136 - val_recall_1: 0.8099
Epoch 22/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0444 - accuracy: 0.9843 - f1_score: 0.9844 - precision_1: 0.9855 - recall_1: 0.9841 - val_loss: 1.2102 - val_accuracy: 0.8166 - val_f1_score: 0.8178 - val_precision_1: 0.8199 - val_recall_1: 0.8144
Epoch 23/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0404 - accuracy: 0.9873 - f1_score: 0.9873 - precision_1: 0.9878 - recall_1: 0.9867 - val_loss: 0.8138 - val_accuracy: 0.8411 - val_f1_score: 0.8436 - val_precision_1: 0.8444 - val_recall_1: 0.8382
Epoch 24/25
379/379 [==============================] - 6s 16ms/step - loss: 0.0300 - accuracy: 0.9900 - f1_score: 0.9900 - precision_1: 0.9904 - recall_1: 0.9899 - val_loss: 0.9011 - val_accuracy: 0.8293 - val_f1_score: 0.8296 - val_precision_1: 0.8342 - val_recall_1: 0.8255
Epoch 25/25
379/379 [==============================] - 7s 17ms/step - loss: 0.0171 - accuracy: 0.9951 - f1_score: 0.9951 - precision_1: 0.9955 - recall_1: 0.9948 - val_loss: 1.0121 - val_accuracy: 0.8419 - val_f1_score: 0.8419 - val_precision_1: 0.8460 - val_recall_1: 0.8404

plot(history)

As we can see in the plot above, there is a problem of early overfitting in the network starting in epoch 5. Thus, in the following section we will apply regularization to check whether it helps to increase the performance of the network.

Test Regularization

In this section several experiments are carried out modifying the previous base architecture:

• Batch normalization layers can be added after the convolutional layers to make the training more stable, allowing a higher learning rate and increasing in turn the speed of the learning process. It also mitigates the drawback of random weight initialization.
• Dropout layers can also be used on the dense layers to eliminate the co-adaptation between them, making the network more robust to possible large individual weights of specific neurons that could dominate among the rest.
• Data augmentation can help to create new synthetic images increasing the number of different samples seen by the network and making the model more robust and independent of the training set.
• Weights regularization restricts the values of the weights to small numbers making their distribution more regular. We will test the two ways of doing this, the L1 regularization and the L2 regularization.

In the following cells we will test several possible combinations of these four elements.

def create_scratch_model(use_batch_normalization=False, use_dropout=False, data_augmentation=None, regularizer=None):
    inputs = keras.Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3))
    
    # (Adjustable) Data augmentation
    if data_augmentation is not None:
        x = data_augmentation(inputs)

    # Feature extraction block
    # Convolution layer
    input_filters = 64
    if regularizer is not None:
        x = layers.Conv2D(input_filters, (3, 3), kernel_regularizer=regularizer, activation="relu")(inputs)
    else:
        x = layers.Conv2D(input_filters, (3, 3), activation="relu")(inputs)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)

    # More convolution layers
    for i in range(3):  # adding variation of layers.
        if regularizer is not None:
            x = layers.Conv2D(input_filters*(2**i),  # increase depth by power of 2 for each conv layer
                              (3, 3), kernel_regularizer=regularizer, activation="relu")(x)
        else:
            x = layers.Conv2D(input_filters*(2**i),  # increase depth by power of 2 for each conv layer
                              (3, 3), activation="relu")(x)
        # (Adjustable) Batch normalization layer
        if use_batch_normalization:
            x = layers.BatchNormalization()(x)
        x = layers.MaxPooling2D(pool_size=(2, 2))(x)

    # Flatten layer
    x = layers.Flatten()(x)

    ##################################################
    # Classification block
    # Fully connected layers
    for i in range(3):
        if regularizer is not None:
            x = layers.Dense(128, kernel_regularizer=regularizer, activation="relu")(x)
        else:
            x = layers.Dense(128, activation="relu")(x)

        # (Adjustable) Dropout layer
        if use_dropout:
            x = layers.Dropout(rate=0.5)(x)
    
    # Output layer
    outputs = layers.Dense(5, activation="softmax")(x)

    # Prepare model
    model = keras.Model(inputs=inputs, outputs=outputs)

    model.compile(optimizer="adam",
        loss="categorical_crossentropy",
        metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
    )

    model.summary()

    return model

def train_model(model, train_dataset, val_dataset, epochs=20, lr=0.001, optimizer=keras.optimizers.Adam, checkpoint=None, lr_scheduler=None, early_stop=None):

  model.compile(optimizer=optimizer(learning_rate=lr),
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
  )

  callbacks = []
  if checkpoint is not None:
    callbacks.append(checkpoint)
  if lr_scheduler is not None:
    callbacks.append(lr_scheduler)
  if early_stop is not None:
    callbacks.append(early_stop)

  history = model.fit(train_dataset, epochs=epochs, validation_data=val_dataset, callbacks=callbacks)
  return history

def execute_pipeline(train_dataset, val_dataset, hyperparameters):

  # Set the seed to ensure reproducibility
  keras.utils.set_random_seed(RANDOM_SEED)

  model = create_scratch_model(use_batch_normalization=hyperparameters["bn"], use_dropout=hyperparameters["dropout"], data_augmentation=hyperparameters["augmentation"], regularizer=hyperparameters["regularizer"])
  history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
  plot(history)

  return model, history

Hyperparameters example

# Example of hyperparameters
hyperparameters = {
    "epochs": 50,  # maximum number of epochs to train the network
    "lr": 0.001,  # initial learning rate of the optimizer
    "optimizer": optimizers.Adam,

    "bn": True,  # if True, perform batch normalization after the convolutional layers
    "dropout": False,  # if True, perform dropout
    
    # Set the following hyperparameters to None if you do not want to use them
    "regularizer":keras.regularizers.l2(0.001),
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + 'checkpoint_name_{epoch}.keras', 'val_loss', save_best_only=True)
}

Experiments

First we will test every regularization technique individually to see how it affects to the resulting training plot.

# Baseline model
checkpoint_filename = "/cnn_baseline_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d (Conv2D)             (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d (MaxPooling2  (None, 89, 89, 64)        0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 43, 43, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_2 (Conv2D)           (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 20, 20, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_3 (Conv2D)           (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 9, 9, 256)         0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 20736)             0         
                                                                 
 dense (Dense)               (None, 128)               2654336   
                                                                 
 dense_1 (Dense)             (None, 128)               16512     
                                                                 
 dense_2 (Dense)             (None, 128)               16512     
                                                                 
 dense_3 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25

2024-03-02 17:28:54.572752: I external/local_xla/xla/service/service.cc:168] XLA service 0x55e57d97a1a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-03-02 17:28:54.572817: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-03-02 17:28:54.605399: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1709396934.878862    3321 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 33s 66ms/step - loss: 1.0156 - accuracy: 0.5700 - f1_score: 0.5714 - precision_1: 0.7555 - recall_1: 0.3995 - val_loss: 0.6200 - val_accuracy: 0.7491 - val_f1_score: 0.7467 - val_precision_1: 0.7929 - val_recall_1: 0.6793
Epoch 2/25
379/379 [==============================] - 20s 53ms/step - loss: 0.5670 - accuracy: 0.7729 - f1_score: 0.7721 - precision_1: 0.8205 - recall_1: 0.7153 - val_loss: 0.5270 - val_accuracy: 0.7832 - val_f1_score: 0.7857 - val_precision_1: 0.8273 - val_recall_1: 0.7327
Epoch 3/25
379/379 [==============================] - 20s 52ms/step - loss: 0.4529 - accuracy: 0.8194 - f1_score: 0.8193 - precision_1: 0.8595 - recall_1: 0.7775 - val_loss: 0.4634 - val_accuracy: 0.8151 - val_f1_score: 0.8151 - val_precision_1: 0.8498 - val_recall_1: 0.7773
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3718 - accuracy: 0.8549 - f1_score: 0.8549 - precision_1: 0.8836 - recall_1: 0.8259 - val_loss: 0.4626 - val_accuracy: 0.8233 - val_f1_score: 0.8234 - val_precision_1: 0.8586 - val_recall_1: 0.7892
Epoch 5/25
379/379 [==============================] - 20s 53ms/step - loss: 0.2977 - accuracy: 0.8826 - f1_score: 0.8825 - precision_1: 0.9024 - recall_1: 0.8612 - val_loss: 0.5014 - val_accuracy: 0.8300 - val_f1_score: 0.8333 - val_precision_1: 0.8470 - val_recall_1: 0.8137
Epoch 6/25
379/379 [==============================] - 20s 52ms/step - loss: 0.2397 - accuracy: 0.9077 - f1_score: 0.9078 - precision_1: 0.9222 - recall_1: 0.8945 - val_loss: 0.4824 - val_accuracy: 0.8241 - val_f1_score: 0.8273 - val_precision_1: 0.8431 - val_recall_1: 0.8137
Epoch 7/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1900 - accuracy: 0.9270 - f1_score: 0.9271 - precision_1: 0.9395 - recall_1: 0.9178 - val_loss: 0.5408 - val_accuracy: 0.8367 - val_f1_score: 0.8386 - val_precision_1: 0.8477 - val_recall_1: 0.8307
Epoch 8/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1639 - accuracy: 0.9385 - f1_score: 0.9385 - precision_1: 0.9461 - recall_1: 0.9318 - val_loss: 0.5398 - val_accuracy: 0.8367 - val_f1_score: 0.8400 - val_precision_1: 0.8445 - val_recall_1: 0.8307
Epoch 9/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1102 - accuracy: 0.9565 - f1_score: 0.9566 - precision_1: 0.9611 - recall_1: 0.9528 - val_loss: 0.6821 - val_accuracy: 0.8248 - val_f1_score: 0.8253 - val_precision_1: 0.8308 - val_recall_1: 0.8203

# Batch normalization model
checkpoint_filename = "/cnn_reg_lr0.001_bn.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": True,
    "dropout": False,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_8"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_10 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_34 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_33 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_35 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 batch_normalization_3 (Bat  (None, 87, 87, 64)        256       
 chNormalization)                                                
                                                                 
 max_pooling2d_34 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_36 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 batch_normalization_4 (Bat  (None, 41, 41, 128)       512       
 chNormalization)                                                
                                                                 
 max_pooling2d_35 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_37 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 batch_normalization_5 (Bat  (None, 18, 18, 256)       1024      
 chNormalization)                                                
                                                                 
 max_pooling2d_36 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_8 (Flatten)         (None, 20736)             0         
                                                                 
 dense_32 (Dense)            (None, 128)               2654336   
                                                                 
 dense_33 (Dense)            (None, 128)               16512     
                                                                 
 dense_34 (Dense)            (None, 128)               16512     
                                                                 
 dense_35 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3097541 (11.82 MB)
Trainable params: 3096645 (11.81 MB)
Non-trainable params: 896 (3.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 32s 77ms/step - loss: 0.9903 - accuracy: 0.6430 - f1_score: 0.6421 - precision_17: 0.6992 - recall_17: 0.5811 - val_loss: 2.1927 - val_accuracy: 0.3920 - val_f1_score: 0.3324 - val_precision_17: 0.4102 - val_recall_17: 0.3474
Epoch 2/25
379/379 [==============================] - 24s 63ms/step - loss: 0.5578 - accuracy: 0.7847 - f1_score: 0.7842 - precision_17: 0.8233 - recall_17: 0.7421 - val_loss: 3.3841 - val_accuracy: 0.3430 - val_f1_score: 0.2958 - val_precision_17: 0.3555 - val_recall_17: 0.3215
Epoch 3/25
379/379 [==============================] - 21s 55ms/step - loss: 0.4481 - accuracy: 0.8290 - f1_score: 0.8288 - precision_17: 0.8576 - recall_17: 0.7993 - val_loss: 0.6664 - val_accuracy: 0.7921 - val_f1_score: 0.7909 - val_precision_17: 0.8166 - val_recall_17: 0.7602
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3647 - accuracy: 0.8565 - f1_score: 0.8566 - precision_17: 0.8790 - recall_17: 0.8323 - val_loss: 0.9244 - val_accuracy: 0.6733 - val_f1_score: 0.6703 - val_precision_17: 0.7060 - val_recall_17: 0.6526
Epoch 5/25
379/379 [==============================] - 20s 52ms/step - loss: 0.2826 - accuracy: 0.8916 - f1_score: 0.8916 - precision_17: 0.9073 - recall_17: 0.8756 - val_loss: 0.8284 - val_accuracy: 0.7105 - val_f1_score: 0.7193 - val_precision_17: 0.7314 - val_recall_17: 0.6852
Epoch 6/25
379/379 [==============================] - 21s 53ms/step - loss: 0.2212 - accuracy: 0.9160 - f1_score: 0.9161 - precision_17: 0.9257 - recall_17: 0.9058 - val_loss: 0.7615 - val_accuracy: 0.7632 - val_f1_score: 0.7690 - val_precision_17: 0.7738 - val_recall_17: 0.7491
Epoch 7/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1833 - accuracy: 0.9309 - f1_score: 0.9310 - precision_17: 0.9385 - recall_17: 0.9256 - val_loss: 2.6059 - val_accuracy: 0.4729 - val_f1_score: 0.4916 - val_precision_17: 0.4841 - val_recall_17: 0.4647
Epoch 8/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1335 - accuracy: 0.9522 - f1_score: 0.9522 - precision_17: 0.9567 - recall_17: 0.9483 - val_loss: 0.6700 - val_accuracy: 0.8203 - val_f1_score: 0.8219 - val_precision_17: 0.8303 - val_recall_17: 0.8137

# Dropout model
checkpoint_filename = "/cnn_reg_lr0.001_dropout.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_3 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_12 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_7 (MaxPoolin  (None, 89, 89, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_13 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_8 (MaxPoolin  (None, 43, 43, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_14 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_9 (MaxPoolin  (None, 20, 20, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_15 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_10 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_2 (Flatten)         (None, 20736)             0         
                                                                 
 dense_6 (Dense)             (None, 128)               2654336   
                                                                 

 dropout_1 (Dropout)         (None, 128)               0         
                                                                 
 dense_7 (Dense)             (None, 128)               16512     
                                                                 
 dropout_2 (Dropout)         (None, 128)               0         
                                                                 
 dense_8 (Dense)             (None, 128)               16512     
                                                                 
 dropout_3 (Dropout)         (None, 128)               0         
                                                                 
 dense_9 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 30s 73ms/step - loss: 1.6085 - accuracy: 0.2067 - f1_score: 0.1912 - precision_5: 0.2973 - recall_5: 9.0707e-04 - val_loss: 1.6097 - val_accuracy: 0.2019 - val_f1_score: 0.0672 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 27s 71ms/step - loss: 1.5623 - accuracy: 0.2670 - f1_score: 0.2524 - precision_5: 0.7171 - recall_5: 0.0297 - val_loss: 1.2725 - val_accuracy: 0.4974 - val_f1_score: 0.4414 - val_precision_5: 0.8279 - val_recall_5: 0.1321
Epoch 3/25
379/379 [==============================] - 27s 70ms/step - loss: 1.0744 - accuracy: 0.5306 - f1_score: 0.5313 - precision_5: 0.7731 - recall_5: 0.2818 - val_loss: 0.8043 - val_accuracy: 0.6273 - val_f1_score: 0.5772 - val_precision_5: 0.8701 - val_recall_5: 0.3927
Epoch 4/25
379/379 [==============================] - 28s 72ms/step - loss: 0.8265 - accuracy: 0.6208 - f1_score: 0.6220 - precision_5: 0.8407 - recall_5: 0.4121 - val_loss: 0.7349 - val_accuracy: 0.6548 - val_f1_score: 0.6168 - val_precision_5: 0.8691 - val_recall_5: 0.4484
Epoch 5/25
379/379 [==============================] - 28s 72ms/step - loss: 0.7364 - accuracy: 0.6643 - f1_score: 0.6637 - precision_5: 0.8545 - recall_5: 0.4701 - val_loss: 0.6555 - val_accuracy: 0.6949 - val_f1_score: 0.6896 - val_precision_5: 0.8883 - val_recall_5: 0.4900
Epoch 6/25
379/379 [==============================] - 28s 72ms/step - loss: 0.6657 - accuracy: 0.7099 - f1_score: 0.7089 - precision_5: 0.8397 - recall_5: 0.5471 - val_loss: 0.6342 - val_accuracy: 0.7357 - val_f1_score: 0.7346 - val_precision_5: 0.8311 - val_recall_5: 0.5880
Epoch 7/25
379/379 [==============================] - 28s 72ms/step - loss: 0.6389 - accuracy: 0.7315 - f1_score: 0.7311 - precision_5: 0.8256 - recall_5: 0.6100 - val_loss: 0.6003 - val_accuracy: 0.7483 - val_f1_score: 0.7514 - val_precision_5: 0.8318 - val_recall_5: 0.6244
Epoch 8/25
379/379 [==============================] - 28s 72ms/step - loss: 0.5575 - accuracy: 0.7708 - f1_score: 0.7707 - precision_5: 0.8441 - recall_5: 0.6828 - val_loss: 0.5500 - val_accuracy: 0.7773 - val_f1_score: 0.7804 - val_precision_5: 0.8220 - val_recall_5: 0.7097
Epoch 9/25
379/379 [==============================] - 26s 67ms/step - loss: 0.4987 - accuracy: 0.8000 - f1_score: 0.7997 - precision_5: 0.8604 - recall_5: 0.7363 - val_loss: 0.5748 - val_accuracy: 0.7765 - val_f1_score: 0.7734 - val_precision_5: 0.8222 - val_recall_5: 0.7313
Epoch 10/25
379/379 [==============================] - 28s 72ms/step - loss: 0.4592 - accuracy: 0.8212 - f1_score: 0.8213 - precision_5: 0.8688 - recall_5: 0.7627 - val_loss: 0.5016 - val_accuracy: 0.8085 - val_f1_score: 0.8096 - val_precision_5: 0.8517 - val_recall_5: 0.7416
Epoch 11/25
379/379 [==============================] - 24s 62ms/step - loss: 0.4354 - accuracy: 0.8343 - f1_score: 0.8344 - precision_5: 0.8790 - recall_5: 0.7826 - val_loss: 0.4808 - val_accuracy: 0.8137 - val_f1_score: 0.8149 - val_precision_5: 0.8588 - val_recall_5: 0.7721
Epoch 12/25
379/379 [==============================] - 21s 54ms/step - loss: 0.3781 - accuracy: 0.8510 - f1_score: 0.8511 - precision_5: 0.8917 - recall_5: 0.8124 - val_loss: 0.5693 - val_accuracy: 0.7899 - val_f1_score: 0.7926 - val_precision_5: 0.8178 - val_recall_5: 0.7565
Epoch 13/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3657 - accuracy: 0.8597 - f1_score: 0.8599 - precision_5: 0.8941 - recall_5: 0.8250 - val_loss: 0.4908 - val_accuracy: 0.8062 - val_f1_score: 0.8087 - val_precision_5: 0.8498 - val_recall_5: 0.7602
Epoch 14/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3470 - accuracy: 0.8733 - f1_score: 0.8735 - precision_5: 0.9037 - recall_5: 0.8396 - val_loss: 0.5385 - val_accuracy: 0.8092 - val_f1_score: 0.8092 - val_precision_5: 0.8462 - val_recall_5: 0.7758
Epoch 15/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3120 - accuracy: 0.8802 - f1_score: 0.8806 - precision_5: 0.9086 - recall_5: 0.8515 - val_loss: 0.5289 - val_accuracy: 0.8077 - val_f1_score: 0.8096 - val_precision_5: 0.8373 - val_recall_5: 0.7869
Epoch 16/25
379/379 [==============================] - 21s 54ms/step - loss: 0.2935 - accuracy: 0.8921 - f1_score: 0.8922 - precision_5: 0.9138 - recall_5: 0.8649 - val_loss: 0.5176 - val_accuracy: 0.8263 - val_f1_score: 0.8287 - val_precision_5: 0.8421 - val_recall_5: 0.7996

# Data augmentation model
checkpoint_filename = "/cnn_reg_lr0.001_da.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": None,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_10"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_12 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_42 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_41 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_43 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_42 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_44 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_43 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_45 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_44 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_10 (Flatten)        (None, 20736)             0         
                                                                 
 dense_40 (Dense)            (None, 128)               2654336   
                                                                 
 dense_41 (Dense)            (None, 128)               16512     
                                                                 
 dense_42 (Dense)            (None, 128)               16512     
                                                                 
 dense_43 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 52ms/step - loss: 0.9927 - accuracy: 0.5770 - f1_score: 0.5758 - precision_21: 0.7633 - recall_21: 0.4055 - val_loss: 0.6403 - val_accuracy: 0.7379 - val_f1_score: 0.7348 - val_precision_21: 0.7946 - val_recall_21: 0.6808
Epoch 2/25
379/379 [==============================] - 20s 52ms/step - loss: 0.5706 - accuracy: 0.7688 - f1_score: 0.7686 - precision_21: 0.8204 - recall_21: 0.7102 - val_loss: 0.5366 - val_accuracy: 0.7810 - val_f1_score: 0.7820 - val_precision_21: 0.8369 - val_recall_21: 0.7201
Epoch 3/25
379/379 [==============================] - 20s 52ms/step - loss: 0.4434 - accuracy: 0.8224 - f1_score: 0.8224 - precision_21: 0.8566 - recall_21: 0.7804 - val_loss: 0.4785 - val_accuracy: 0.8077 - val_f1_score: 0.8083 - val_precision_21: 0.8341 - val_recall_21: 0.7840
Epoch 4/25
379/379 [==============================] - 20s 52ms/step - loss: 0.3522 - accuracy: 0.8621 - f1_score: 0.8622 - precision_21: 0.8862 - recall_21: 0.8319 - val_loss: 0.5314 - val_accuracy: 0.8085 - val_f1_score: 0.8069 - val_precision_21: 0.8258 - val_recall_21: 0.7847
Epoch 5/25
379/379 [==============================] - 20s 52ms/step - loss: 0.2948 - accuracy: 0.8869 - f1_score: 0.8871 - precision_21: 0.9062 - recall_21: 0.8680 - val_loss: 0.5559 - val_accuracy: 0.8129 - val_f1_score: 0.8130 - val_precision_21: 0.8336 - val_recall_21: 0.7958
Epoch 6/25
379/379 [==============================] - 20s 52ms/step - loss: 0.2228 - accuracy: 0.9119 - f1_score: 0.9120 - precision_21: 0.9277 - recall_21: 0.8987 - val_loss: 0.5561 - val_accuracy: 0.8211 - val_f1_score: 0.8238 - val_precision_21: 0.8362 - val_recall_21: 0.8070
Epoch 7/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1655 - accuracy: 0.9358 - f1_score: 0.9359 - precision_21: 0.9444 - recall_21: 0.9278 - val_loss: 0.6472 - val_accuracy: 0.8226 - val_f1_score: 0.8230 - val_precision_21: 0.8342 - val_recall_21: 0.8181
Epoch 8/25
379/379 [==============================] - 20s 52ms/step - loss: 0.1426 - accuracy: 0.9478 - f1_score: 0.9479 - precision_21: 0.9535 - recall_21: 0.9416 - val_loss: 0.6518 - val_accuracy: 0.8293 - val_f1_score: 0.8308 - val_precision_21: 0.8367 - val_recall_21: 0.8255

# L1 model
checkpoint_filename = "/cnn_reg_lr0.001_l1.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": keras.regularizers.l1(0.001),
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_11"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_13 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_46 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_45 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_47 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_46 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_48 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_47 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_49 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_48 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_11 (Flatten)        (None, 20736)             0         
                                                                 
 dense_44 (Dense)            (None, 128)               2654336   
                                                                 
 dense_45 (Dense)            (None, 128)               16512     
                                                                 
 dense_46 (Dense)            (None, 128)               16512     
                                                                 
 dense_47 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 52ms/step - loss: 3.3340 - accuracy: 0.1982 - f1_score: 0.1396 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9881 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 20s 52ms/step - loss: 1.9908 - accuracy: 0.2001 - f1_score: 0.1109 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9919 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 21s 53ms/step - loss: 1.9909 - accuracy: 0.1984 - f1_score: 0.0993 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9830 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 20s 53ms/step - loss: 1.9909 - accuracy: 0.1982 - f1_score: 0.0967 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9894 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 20s 52ms/step - loss: 1.9909 - accuracy: 0.2005 - f1_score: 0.1098 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9908 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 20s 52ms/step - loss: 1.9909 - accuracy: 0.1993 - f1_score: 0.0952 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9937 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 20s 53ms/step - loss: 1.9909 - accuracy: 0.1991 - f1_score: 0.0976 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9987 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 20s 52ms/step - loss: 1.9909 - accuracy: 0.1991 - f1_score: 0.1056 - precision_23: 0.0000e+00 - recall_23: 0.0000e+00 - val_loss: 1.9963 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_23: 0.0000e+00 - val_recall_23: 0.0000e+00

# L2 model
checkpoint_filename = "/cnn_reg_lr0.001_l2.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": keras.regularizers.l2(0.001),
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_7 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_22 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_21 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_23 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_22 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_24 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_23 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_25 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_24 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_5 (Flatten)         (None, 20736)             0         
                                                                 
 dense_20 (Dense)            (None, 128)               2654336   
                                                                 
 dense_21 (Dense)            (None, 128)               16512     
                                                                 
 dense_22 (Dense)            (None, 128)               16512     
                                                                 
 dense_23 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 23s 54ms/step - loss: 1.5102 - accuracy: 0.4042 - f1_score: 0.4040 - precision_11: 0.6437 - recall_11: 0.1455 - val_loss: 1.0197 - val_accuracy: 0.5664 - val_f1_score: 0.5266 - val_precision_11: 0.7821 - val_recall_11: 0.3623 - lr: 0.0010
Epoch 2/25
379/379 [==============================] - 21s 54ms/step - loss: 0.9108 - accuracy: 0.6605 - f1_score: 0.6589 - precision_11: 0.7620 - recall_11: 0.5214 - val_loss: 0.8186 - val_accuracy: 0.7164 - val_f1_score: 0.7235 - val_precision_11: 0.7937 - val_recall_11: 0.5768 - lr: 0.0010
Epoch 3/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7332 - accuracy: 0.7398 - f1_score: 0.7393 - precision_11: 0.7970 - recall_11: 0.6640 - val_loss: 0.7253 - val_accuracy: 0.7461 - val_f1_score: 0.7466 - val_precision_11: 0.8007 - val_recall_11: 0.6741 - lr: 0.0010
Epoch 4/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6831 - accuracy: 0.7584 - f1_score: 0.7578 - precision_11: 0.8150 - recall_11: 0.7006 - val_loss: 0.7004 - val_accuracy: 0.7483 - val_f1_score: 0.7456 - val_precision_11: 0.8075 - val_recall_11: 0.6882 - lr: 0.0010
Epoch 5/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6277 - accuracy: 0.7833 - f1_score: 0.7827 - precision_11: 0.8312 - recall_11: 0.7287 - val_loss: 0.6670 - val_accuracy: 0.7684 - val_f1_score: 0.7643 - val_precision_11: 0.8204 - val_recall_11: 0.7090 - lr: 0.0010
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6118 - accuracy: 0.7955 - f1_score: 0.7952 - precision_11: 0.8379 - recall_11: 0.7474 - val_loss: 0.6605 - val_accuracy: 0.7780 - val_f1_score: 0.7793 - val_precision_11: 0.8126 - val_recall_11: 0.7305 - lr: 0.0010
Epoch 7/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5758 - accuracy: 0.8109 - f1_score: 0.8109 - precision_11: 0.8465 - recall_11: 0.7692 - val_loss: 0.6411 - val_accuracy: 0.7706 - val_f1_score: 0.7747 - val_precision_11: 0.8263 - val_recall_11: 0.7313 - lr: 0.0010
Epoch 8/25
379/379 [==============================] - 21s 53ms/step - loss: 0.5481 - accuracy: 0.8216 - f1_score: 0.8215 - precision_11: 0.8576 - recall_11: 0.7866 - val_loss: 0.6655 - val_accuracy: 0.7751 - val_f1_score: 0.7698 - val_precision_11: 0.8109 - val_recall_11: 0.7387 - lr: 0.0010
Epoch 9/25
379/379 [==============================] - 21s 53ms/step - loss: 0.5457 - accuracy: 0.8249 - f1_score: 0.8249 - precision_11: 0.8614 - recall_11: 0.7892 - val_loss: 0.6446 - val_accuracy: 0.7921 - val_f1_score: 0.7961 - val_precision_11: 0.8168 - val_recall_11: 0.7647 - lr: 0.0010
Epoch 10/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5101 - accuracy: 0.8448 - f1_score: 0.8449 - precision_11: 0.8737 - recall_11: 0.8126 - val_loss: 0.6359 - val_accuracy: 0.7981 - val_f1_score: 0.7971 - val_precision_11: 0.8252 - val_recall_11: 0.7743 - lr: 0.0010
Epoch 11/25
379/379 [==============================] - 21s 53ms/step - loss: 0.4895 - accuracy: 0.8573 - f1_score: 0.8574 - precision_11: 0.8824 - recall_11: 0.8305 - val_loss: 0.6492 - val_accuracy: 0.7936 - val_f1_score: 0.7947 - val_precision_11: 0.8240 - val_recall_11: 0.7684 - lr: 0.0010
Epoch 12/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4627 - accuracy: 0.8703 - f1_score: 0.8704 - precision_11: 0.8922 - recall_11: 0.8470 - val_loss: 0.6247 - val_accuracy: 0.8114 - val_f1_score: 0.8162 - val_precision_11: 0.8340 - val_recall_11: 0.7869 - lr: 0.0010
Epoch 13/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4455 - accuracy: 0.8797 - f1_score: 0.8798 - precision_11: 0.8999 - recall_11: 0.8601 - val_loss: 0.6924 - val_accuracy: 0.7847 - val_f1_score: 0.7854 - val_precision_11: 0.8157 - val_recall_11: 0.7691 - lr: 0.0010
Epoch 14/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4258 - accuracy: 0.8871 - f1_score: 0.8872 - precision_11: 0.9058 - recall_11: 0.8693 - val_loss: 0.6670 - val_accuracy: 0.8025 - val_f1_score: 0.8019 - val_precision_11: 0.8266 - val_recall_11: 0.7788 - lr: 0.0010
Epoch 15/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4127 - accuracy: 0.8954 - f1_score: 0.8956 - precision_11: 0.9123 - recall_11: 0.8787 - val_loss: 0.6830 - val_accuracy: 0.8085 - val_f1_score: 0.8122 - val_precision_11: 0.8255 - val_recall_11: 0.7869 - lr: 0.0010
Epoch 16/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4093 - accuracy: 0.9013 - f1_score: 0.9014 - precision_11: 0.9164 - recall_11: 0.8865 - val_loss: 0.6769 - val_accuracy: 0.8159 - val_f1_score: 0.8170 - val_precision_11: 0.8333 - val_recall_11: 0.8018 - lr: 0.0010
Epoch 17/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3794 - accuracy: 0.9132 - f1_score: 0.9133 - precision_11: 0.9253 - recall_11: 0.9017 - val_loss: 0.6959 - val_accuracy: 0.8025 - val_f1_score: 0.8059 - val_precision_11: 0.8227 - val_recall_11: 0.7788 - lr: 0.0010

# L1_L2 model
checkpoint_filename = "/cnn_reg_lr0.001_l1_l2.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": keras.regularizers.l1_l2(l1=0.001, l2=0.001),
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_6"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_8 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_26 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_25 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_27 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_26 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_28 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_27 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_29 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_28 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_6 (Flatten)         (None, 20736)             0         
                                                                 
 dense_24 (Dense)            (None, 128)               2654336   
                                                                 
 dense_25 (Dense)            (None, 128)               16512     
                                                                 
 dense_26 (Dense)            (None, 128)               16512     
                                                                 
 dense_27 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 23s 54ms/step - loss: 3.4400 - accuracy: 0.1982 - f1_score: 0.1372 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9841 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 2/25
379/379 [==============================] - 21s 55ms/step - loss: 1.9850 - accuracy: 0.1972 - f1_score: 0.1250 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9835 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 3/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9792 - accuracy: 0.1996 - f1_score: 0.0964 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9775 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 4/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9825 - accuracy: 0.1997 - f1_score: 0.0963 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9845 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 5/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9835 - accuracy: 0.1975 - f1_score: 0.1090 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9829 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9894 - accuracy: 0.2000 - f1_score: 0.0920 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9946 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 7/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9951 - accuracy: 0.1991 - f1_score: 0.1050 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9958 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 1.9955 - accuracy: 0.2000 - f1_score: 0.0960 - precision_13: 0.0000e+00 - recall_13: 0.0000e+00 - val_loss: 1.9964 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_13: 0.0000e+00 - val_recall_13: 0.0000e+00 - lr: 0.0010

# Learning rate scheduler model
checkpoint_filename = "/cnn_reg_lr0.001_lrScheduler.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_14 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_50 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_49 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_51 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_50 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_52 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_51 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_53 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_52 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_12 (Flatten)        (None, 20736)             0         
                                                                 
 dense_48 (Dense)            (None, 128)               2654336   
                                                                 
 dense_49 (Dense)            (None, 128)               16512     
                                                                 
 dense_50 (Dense)            (None, 128)               16512     
                                                                 
 dense_51 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 52ms/step - loss: 0.9525 - accuracy: 0.5894 - f1_score: 0.5881 - precision_25: 0.7719 - recall_25: 0.4149 - val_loss: 0.6880 - val_accuracy: 0.7201 - val_f1_score: 0.7141 - val_precision_25: 0.7862 - val_recall_25: 0.6578 - lr: 0.0010
Epoch 2/25
379/379 [==============================] - 20s 52ms/step - loss: 0.5696 - accuracy: 0.7686 - f1_score: 0.7684 - precision_25: 0.8188 - recall_25: 0.7107 - val_loss: 0.5281 - val_accuracy: 0.7780 - val_f1_score: 0.7808 - val_precision_25: 0.8159 - val_recall_25: 0.7402 - lr: 0.0010
Epoch 3/25
379/379 [==============================] - 21s 53ms/step - loss: 0.4572 - accuracy: 0.8195 - f1_score: 0.8193 - precision_25: 0.8538 - recall_25: 0.7790 - val_loss: 0.4642 - val_accuracy: 0.8077 - val_f1_score: 0.8078 - val_precision_25: 0.8266 - val_recall_25: 0.7854 - lr: 0.0010
Epoch 4/25
379/379 [==============================] - 20s 52ms/step - loss: 0.3714 - accuracy: 0.8547 - f1_score: 0.8547 - precision_25: 0.8813 - recall_25: 0.8270 - val_loss: 0.5008 - val_accuracy: 0.7936 - val_f1_score: 0.7913 - val_precision_25: 0.8190 - val_recall_25: 0.7758 - lr: 0.0010
Epoch 5/25
379/379 [==============================] - 20s 52ms/step - loss: 0.2928 - accuracy: 0.8842 - f1_score: 0.8844 - precision_25: 0.9058 - recall_25: 0.8672 - val_loss: 0.5287 - val_accuracy: 0.8040 - val_f1_score: 0.8029 - val_precision_25: 0.8239 - val_recall_25: 0.7743 - lr: 0.0010
Epoch 6/25
379/379 [==============================] - 21s 55ms/step - loss: 0.2427 - accuracy: 0.9050 - f1_score: 0.9052 - precision_25: 0.9196 - recall_25: 0.8917 - val_loss: 0.4806 - val_accuracy: 0.8189 - val_f1_score: 0.8182 - val_precision_25: 0.8421 - val_recall_25: 0.8040 - lr: 0.0010
Epoch 7/25
379/379 [==============================] - 22s 57ms/step - loss: 0.1726 - accuracy: 0.9341 - f1_score: 0.9343 - precision_25: 0.9434 - recall_25: 0.9262 - val_loss: 0.5333 - val_accuracy: 0.8500 - val_f1_score: 0.8514 - val_precision_25: 0.8576 - val_recall_25: 0.8404 - lr: 0.0010
Epoch 8/25
379/379 [==============================] - 22s 56ms/step - loss: 0.1355 - accuracy: 0.9474 - f1_score: 0.9474 - precision_25: 0.9528 - recall_25: 0.9421 - val_loss: 0.6452 - val_accuracy: 0.8196 - val_f1_score: 0.8205 - val_precision_25: 0.8243 - val_recall_25: 0.8114 - lr: 0.0010

First of all, we have tried using batch normalization layers after the convolutional layers to make the training faster and more stable. However, it did not seem to reduce overfitting and even affected the loss curve making the training much more inestable. On the other hand, using dropout and data normalization seemed to affect positively to training, specially the former one which achieved to delay the overfitting emergence to the epoch 7 and obtaining a validation accuracy of ~0.82, slighly lower than the baseline model.

Regarding weight regularization, L1 regularization makes the model completely unable to learn the patterns from the dataset, returning a validacion_accuracy of 0.2 (random classification since there are 5 classes). L1 regularization is generally considered to generate models that are simple and interpretable but not capable of learning complex patterns, which highlights the necesity of learning complex patterns in our dataset. The same occurs when combined with L2.

However, applying L2 regularization we achieve to stabilize the training delaying the overfitting emergence until epoch 12 and obtaining a validation accuracy of 0.8159.

Let's try now combining some of the techniques which seemed to deal with the overfitting problem. First, we will combine Dropout with Data augmentation and see what happens.

# Dropout augmented model
checkpoint_filename = "/cnn_reg_lr0.001_dropout_aug.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_4 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_16 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_11 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_17 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_12 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_18 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_13 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_19 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_14 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_3 (Flatten)         (None, 20736)             0         
                                                                 
 dense_10 (Dense)            (None, 128)               2654336   
                                                                 
 dropout_4 (Dropout)         (None, 128)               0         
                                                                 
 dense_11 (Dense)            (None, 128)               16512     
                                                                 
 dropout_5 (Dropout)         (None, 128)               0         
                                                                 
 dense_12 (Dense)            (None, 128)               16512     
                                                                 
 dropout_6 (Dropout)         (None, 128)               0         
                                                                 
 dense_13 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 53ms/step - loss: 1.6117 - accuracy: 0.2010 - f1_score: 0.1852 - precision_7: 0.1667 - recall_7: 8.2461e-05 - val_loss: 1.6097 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6103 - accuracy: 0.1977 - f1_score: 0.1676 - precision_7: 0.0000e+00 - recall_7: 0.0000e+00 - val_loss: 1.6096 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6100 - accuracy: 0.2039 - f1_score: 0.1674 - precision_7: 0.0000e+00 - recall_7: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 20s 52ms/step - loss: 1.6110 - accuracy: 0.2080 - f1_score: 0.1861 - precision_7: 0.3333 - recall_7: 4.9476e-04 - val_loss: 1.6095 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6095 - accuracy: 0.2017 - f1_score: 0.1696 - precision_7: 0.3333 - recall_7: 4.1230e-04 - val_loss: 1.6095 - val_accuracy: 0.2064 - val_f1_score: 0.0684 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6102 - accuracy: 0.2024 - f1_score: 0.1729 - precision_7: 0.2500 - recall_7: 8.2461e-05 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 20s 53ms/step - loss: 1.6098 - accuracy: 0.2030 - f1_score: 0.1637 - precision_7: 0.0000e+00 - recall_7: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6096 - accuracy: 0.2070 - f1_score: 0.1517 - precision_7: 0.0000e+00 - recall_7: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_7: 0.0000e+00 - val_recall_7: 0.0000e+00

As we can see in the plot above, the model updates the weights too drastically causing a inestability in the training. This situation seems to be caused by a too high learning rate. Below we will test what happens if we vary this parameter.

# Dropout augmented model, with higher learning rate
checkpoint_filename = "/cnn_reg_lr0.01_dropout_aug.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.01,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_5 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_20 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_15 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_21 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_16 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_22 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_17 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_23 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_18 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_4 (Flatten)         (None, 20736)             0         
                                                                 
 dense_14 (Dense)            (None, 128)               2654336   
                                                                 
 dropout_7 (Dropout)         (None, 128)               0         
                                                                 
 dense_15 (Dense)            (None, 128)               16512     
                                                                 
 dropout_8 (Dropout)         (None, 128)               0         
                                                                 
 dense_16 (Dense)            (None, 128)               16512     
                                                                 
 dropout_9 (Dropout)         (None, 128)               0         
                                                                 
 dense_17 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 53ms/step - loss: 2.2046 - accuracy: 0.2029 - f1_score: 0.2001 - precision_9: 0.2459 - recall_9: 0.0012 - val_loss: 1.6123 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 21s 54ms/step - loss: 1.6111 - accuracy: 0.1962 - f1_score: 0.1831 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6116 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6108 - accuracy: 0.1954 - f1_score: 0.1769 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6121 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6108 - accuracy: 0.1972 - f1_score: 0.1799 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6107 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 25s 63ms/step - loss: 1.6107 - accuracy: 0.1996 - f1_score: 0.1748 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6115 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 28s 71ms/step - loss: 1.6108 - accuracy: 0.1907 - f1_score: 0.1701 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6114 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 27s 71ms/step - loss: 1.6108 - accuracy: 0.1920 - f1_score: 0.1656 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6111 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 28s 71ms/step - loss: 1.6108 - accuracy: 0.1911 - f1_score: 0.1769 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6113 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00
Epoch 9/25
379/379 [==============================] - 21s 53ms/step - loss: 1.6107 - accuracy: 0.1966 - f1_score: 0.1718 - precision_9: 0.0000e+00 - recall_9: 0.0000e+00 - val_loss: 1.6110 - val_accuracy: 0.1893 - val_f1_score: 0.0637 - val_precision_9: 0.0000e+00 - val_recall_9: 0.0000e+00

# Dropout augmented model, with lower learning rate
checkpoint_filename = "/cnn_reg_lr0.0005_dropout_aug.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0005,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_6 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_24 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_19 (MaxPooli  (None, 89, 89, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_25 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_20 (MaxPooli  (None, 43, 43, 64)        0         
 ng2D)                                                           
                                                                 
 conv2d_26 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_21 (MaxPooli  (None, 20, 20, 128)       0         
 ng2D)                                                           
                                                                 
 conv2d_27 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_22 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_5 (Flatten)         (None, 20736)             0         
                                                                 
 dense_18 (Dense)            (None, 128)               2654336   
                                                                 
 dropout_10 (Dropout)        (None, 128)               0         
                                                                 
 dense_19 (Dense)            (None, 128)               16512     
                                                                 
 dropout_11 (Dropout)        (None, 128)               0         
                                                                 
 dense_20 (Dense)            (None, 128)               16512     
                                                                 
 dropout_12 (Dropout)        (None, 128)               0         
                                                                 
 dense_21 (Dense)            (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 22s 53ms/step - loss: 1.4324 - accuracy: 0.3499 - f1_score: 0.3459 - precision_11: 0.5737 - recall_11: 0.0806 - val_loss: 0.9033 - val_accuracy: 0.6407 - val_f1_score: 0.6318 - val_precision_11: 0.8117 - val_recall_11: 0.3808
Epoch 2/25
379/379 [==============================] - 21s 53ms/step - loss: 0.9208 - accuracy: 0.5982 - f1_score: 0.5962 - precision_11: 0.7460 - recall_11: 0.4121 - val_loss: 0.6936 - val_accuracy: 0.7008 - val_f1_score: 0.7074 - val_precision_11: 0.8770 - val_recall_11: 0.4870
Epoch 3/25
379/379 [==============================] - 21s 53ms/step - loss: 0.7419 - accuracy: 0.6880 - f1_score: 0.6871 - precision_11: 0.8104 - recall_11: 0.5300 - val_loss: 0.5929 - val_accuracy: 0.7572 - val_f1_score: 0.7596 - val_precision_11: 0.9030 - val_recall_11: 0.5947
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 0.6516 - accuracy: 0.7351 - f1_score: 0.7349 - precision_11: 0.8359 - recall_11: 0.6082 - val_loss: 0.5565 - val_accuracy: 0.7854 - val_f1_score: 0.7853 - val_precision_11: 0.8830 - val_recall_11: 0.6837
Epoch 5/25
379/379 [==============================] - 21s 53ms/step - loss: 0.5795 - accuracy: 0.7694 - f1_score: 0.7694 - precision_11: 0.8489 - recall_11: 0.6715 - val_loss: 0.4477 - val_accuracy: 0.8211 - val_f1_score: 0.8243 - val_precision_11: 0.8688 - val_recall_11: 0.7669
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5027 - accuracy: 0.8056 - f1_score: 0.8057 - precision_11: 0.8656 - recall_11: 0.7339 - val_loss: 0.4350 - val_accuracy: 0.8315 - val_f1_score: 0.8335 - val_precision_11: 0.8781 - val_recall_11: 0.7699
Epoch 7/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4552 - accuracy: 0.8275 - f1_score: 0.8275 - precision_11: 0.8806 - recall_11: 0.7647 - val_loss: 0.3990 - val_accuracy: 0.8426 - val_f1_score: 0.8416 - val_precision_11: 0.8814 - val_recall_11: 0.7944
Epoch 8/25
379/379 [==============================] - 21s 53ms/step - loss: 0.4015 - accuracy: 0.8495 - f1_score: 0.8496 - precision_11: 0.8906 - recall_11: 0.7991 - val_loss: 0.4157 - val_accuracy: 0.8285 - val_f1_score: 0.8324 - val_precision_11: 0.8655 - val_recall_11: 0.7929
Epoch 9/25
379/379 [==============================] - 21s 54ms/step - loss: 0.3663 - accuracy: 0.8656 - f1_score: 0.8656 - precision_11: 0.9019 - recall_11: 0.8238 - val_loss: 0.3822 - val_accuracy: 0.8552 - val_f1_score: 0.8568 - val_precision_11: 0.8795 - val_recall_11: 0.8293
Epoch 10/25
379/379 [==============================] - 21s 54ms/step - loss: 0.3282 - accuracy: 0.8774 - f1_score: 0.8776 - precision_11: 0.9072 - recall_11: 0.8446 - val_loss: 0.3759 - val_accuracy: 0.8552 - val_f1_score: 0.8574 - val_precision_11: 0.8807 - val_recall_11: 0.8389
Epoch 11/25
379/379 [==============================] - 21s 54ms/step - loss: 0.2955 - accuracy: 0.8923 - f1_score: 0.8925 - precision_11: 0.9178 - recall_11: 0.8620 - val_loss: 0.3831 - val_accuracy: 0.8478 - val_f1_score: 0.8509 - val_precision_11: 0.8656 - val_recall_11: 0.8322
Epoch 12/25
379/379 [==============================] - 21s 55ms/step - loss: 0.2637 - accuracy: 0.9014 - f1_score: 0.9015 - precision_11: 0.9223 - recall_11: 0.8785 - val_loss: 0.3767 - val_accuracy: 0.8604 - val_f1_score: 0.8634 - val_precision_11: 0.8739 - val_recall_11: 0.8486
Epoch 13/25
379/379 [==============================] - 21s 55ms/step - loss: 0.2532 - accuracy: 0.9109 - f1_score: 0.9111 - precision_11: 0.9311 - recall_11: 0.8908 - val_loss: 0.4229 - val_accuracy: 0.8552 - val_f1_score: 0.8567 - val_precision_11: 0.8709 - val_recall_11: 0.8463
Epoch 14/25
379/379 [==============================] - 21s 53ms/step - loss: 0.2071 - accuracy: 0.9270 - f1_score: 0.9272 - precision_11: 0.9411 - recall_11: 0.9124 - val_loss: 0.4040 - val_accuracy: 0.8575 - val_f1_score: 0.8585 - val_precision_11: 0.8783 - val_recall_11: 0.8411
Epoch 15/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1927 - accuracy: 0.9322 - f1_score: 0.9323 - precision_11: 0.9461 - recall_11: 0.9201 - val_loss: 0.4419 - val_accuracy: 0.8463 - val_f1_score: 0.8463 - val_precision_11: 0.8654 - val_recall_11: 0.8352

As we expected, the learning rate was too high. Reducing it to 0.0005 allowed to stabilize the training and obtain a ~0.86 validation accuracy avoiding early overfitting problems as well. Having as a goal to reach the 90% in validation accuracy, we decided to repeat the previous process adding a learning rate scheduler to progressively reduce the learning rate value (Learning rate decay) and then test adding L2 regularization which provided good results in the previous experiments.

# Dropout augmented model with learning rate scheduler
checkpoint_filename = "/cnn_reg_lr0.0005_dropout_aug_lrScheduler.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0005,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_2 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_8 (Conv2D)           (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 89, 89, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_9 (Conv2D)           (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_4 (MaxPoolin  (None, 43, 43, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_10 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_5 (MaxPoolin  (None, 20, 20, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_11 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_6 (MaxPoolin  (None, 9, 9, 256)         0         
 g2D)                                                            
                                                                 
 flatten_1 (Flatten)         (None, 20736)             0         
                                                                 
 dense_2 (Dense)             (None, 128)               2654336   
                                                                 
 dropout_1 (Dropout)         (None, 128)               0         
                                                                 
 dense_3 (Dense)             (None, 128)               16512     
                                                                 
 dropout_2 (Dropout)         (None, 128)               0         
                                                                 
 dense_4 (Dense)             (None, 128)               16512     
                                                                 
 dropout_3 (Dropout)         (None, 128)               0         
                                                                 
 dense_5 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 41s 87ms/step - loss: 1.4127 - accuracy: 0.3618 - f1_score: 0.3607 - precision_3: 0.6615 - recall_3: 0.1028 - val_loss: 0.9307 - val_accuracy: 0.5961 - val_f1_score: 0.5770 - val_precision_3: 0.7765 - val_recall_3: 0.3638 - lr: 5.0000e-04
Epoch 2/25
379/379 [==============================] - 28s 73ms/step - loss: 0.9141 - accuracy: 0.6111 - f1_score: 0.6106 - precision_3: 0.7650 - recall_3: 0.4086 - val_loss: 0.7101 - val_accuracy: 0.7120 - val_f1_score: 0.7141 - val_precision_3: 0.8755 - val_recall_3: 0.4699 - lr: 5.0000e-04
Epoch 3/25
379/379 [==============================] - 28s 73ms/step - loss: 0.7166 - accuracy: 0.6961 - f1_score: 0.6956 - precision_3: 0.8080 - recall_3: 0.5458 - val_loss: 0.5835 - val_accuracy: 0.7728 - val_f1_score: 0.7771 - val_precision_3: 0.8698 - val_recall_3: 0.6347 - lr: 5.0000e-04
Epoch 4/25
379/379 [==============================] - 28s 73ms/step - loss: 0.6280 - accuracy: 0.7505 - f1_score: 0.7500 - precision_3: 0.8358 - recall_3: 0.6348 - val_loss: 0.4871 - val_accuracy: 0.8040 - val_f1_score: 0.8060 - val_precision_3: 0.8712 - val_recall_3: 0.7431 - lr: 5.0000e-04
Epoch 5/25
379/379 [==============================] - 28s 73ms/step - loss: 0.5375 - accuracy: 0.7877 - f1_score: 0.7875 - precision_3: 0.8561 - recall_3: 0.7043 - val_loss: 0.4317 - val_accuracy: 0.8300 - val_f1_score: 0.8328 - val_precision_3: 0.8618 - val_recall_3: 0.7825 - lr: 5.0000e-04
Epoch 6/25
379/379 [==============================] - 28s 73ms/step - loss: 0.4602 - accuracy: 0.8236 - f1_score: 0.8237 - precision_3: 0.8743 - recall_3: 0.7624 - val_loss: 0.4343 - val_accuracy: 0.8278 - val_f1_score: 0.8321 - val_precision_3: 0.8529 - val_recall_3: 0.7966 - lr: 5.0000e-04
Epoch 7/25
379/379 [==============================] - 28s 73ms/step - loss: 0.4199 - accuracy: 0.8419 - f1_score: 0.8420 - precision_3: 0.8844 - recall_3: 0.7916 - val_loss: 0.3949 - val_accuracy: 0.8448 - val_f1_score: 0.8479 - val_precision_3: 0.8720 - val_recall_3: 0.8092 - lr: 5.0000e-04
Epoch 8/25
379/379 [==============================] - 28s 73ms/step - loss: 0.3709 - accuracy: 0.8594 - f1_score: 0.8596 - precision_3: 0.8987 - recall_3: 0.8187 - val_loss: 0.3958 - val_accuracy: 0.8426 - val_f1_score: 0.8448 - val_precision_3: 0.8714 - val_recall_3: 0.8248 - lr: 5.0000e-04
Epoch 9/25
379/379 [==============================] - 28s 73ms/step - loss: 0.3363 - accuracy: 0.8751 - f1_score: 0.8753 - precision_3: 0.9076 - recall_3: 0.8409 - val_loss: 0.4284 - val_accuracy: 0.8389 - val_f1_score: 0.8400 - val_precision_3: 0.8668 - val_recall_3: 0.8114 - lr: 5.0000e-04
Epoch 10/25
379/379 [==============================] - 28s 72ms/step - loss: 0.3197 - accuracy: 0.8797 - f1_score: 0.8800 - precision_3: 0.9086 - recall_3: 0.8488 - val_loss: 0.3758 - val_accuracy: 0.8567 - val_f1_score: 0.8594 - val_precision_3: 0.8735 - val_recall_3: 0.8359 - lr: 5.0000e-04
Epoch 11/25
379/379 [==============================] - 28s 73ms/step - loss: 0.2672 - accuracy: 0.9003 - f1_score: 0.9005 - precision_3: 0.9187 - recall_3: 0.8785 - val_loss: 0.4098 - val_accuracy: 0.8649 - val_f1_score: 0.8654 - val_precision_3: 0.8834 - val_recall_3: 0.8441 - lr: 5.0000e-04
Epoch 12/25
379/379 [==============================] - 28s 73ms/step - loss: 0.2616 - accuracy: 0.9038 - f1_score: 0.9040 - precision_3: 0.9239 - recall_3: 0.8847 - val_loss: 0.4321 - val_accuracy: 0.8493 - val_f1_score: 0.8517 - val_precision_3: 0.8664 - val_recall_3: 0.8330 - lr: 5.0000e-04
Epoch 13/25
379/379 [==============================] - 28s 73ms/step - loss: 0.2349 - accuracy: 0.9163 - f1_score: 0.9165 - precision_3: 0.9330 - recall_3: 0.8992 - val_loss: 0.3680 - val_accuracy: 0.8738 - val_f1_score: 0.8758 - val_precision_3: 0.8849 - val_recall_3: 0.8619 - lr: 5.0000e-04
Epoch 14/25
379/379 [==============================] - 28s 73ms/step - loss: 0.2118 - accuracy: 0.9250 - f1_score: 0.9251 - precision_3: 0.9380 - recall_3: 0.9109 - val_loss: 0.3846 - val_accuracy: 0.8693 - val_f1_score: 0.8714 - val_precision_3: 0.8838 - val_recall_3: 0.8582 - lr: 5.0000e-04
Epoch 15/25
379/379 [==============================] - 26s 68ms/step - loss: 0.1889 - accuracy: 0.9321 - f1_score: 0.9323 - precision_3: 0.9469 - recall_3: 0.9196 - val_loss: 0.4029 - val_accuracy: 0.8716 - val_f1_score: 0.8723 - val_precision_3: 0.8875 - val_recall_3: 0.8552 - lr: 5.0000e-04
Epoch 16/25
379/379 [==============================] - 22s 56ms/step - loss: 0.1787 - accuracy: 0.9376 - f1_score: 0.9377 - precision_3: 0.9489 - recall_3: 0.9245 - val_loss: 0.3703 - val_accuracy: 0.8708 - val_f1_score: 0.8722 - val_precision_3: 0.8820 - val_recall_3: 0.8604 - lr: 5.0000e-04
Epoch 17/25
379/379 [==============================] - 26s 67ms/step - loss: 0.1783 - accuracy: 0.9402 - f1_score: 0.9404 - precision_3: 0.9511 - recall_3: 0.9294 - val_loss: 0.4008 - val_accuracy: 0.8753 - val_f1_score: 0.8759 - val_precision_3: 0.8862 - val_recall_3: 0.8612 - lr: 5.0000e-04
Epoch 18/25
379/379 [==============================] - 26s 68ms/step - loss: 0.1435 - accuracy: 0.9499 - f1_score: 0.9500 - precision_3: 0.9578 - recall_3: 0.9404 - val_loss: 0.4684 - val_accuracy: 0.8575 - val_f1_score: 0.8596 - val_precision_3: 0.8714 - val_recall_3: 0.8500 - lr: 5.0000e-04

# Dropout augmented model with l2 and learning rate scheduler
checkpoint_filename = "/cnn_reg_lr0.0005_dropout_aug_l2_lrScheduler.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0005,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": keras.regularizers.l2(0.001),
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_3 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_12 (Conv2D)          (None, 178, 178, 64)      1792      
                                                                 
 max_pooling2d_7 (MaxPoolin  (None, 89, 89, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_13 (Conv2D)          (None, 87, 87, 64)        36928     
                                                                 
 max_pooling2d_8 (MaxPoolin  (None, 43, 43, 64)        0         
 g2D)                                                            
                                                                 
 conv2d_14 (Conv2D)          (None, 41, 41, 128)       73856     
                                                                 
 max_pooling2d_9 (MaxPoolin  (None, 20, 20, 128)       0         
 g2D)                                                            
                                                                 
 conv2d_15 (Conv2D)          (None, 18, 18, 256)       295168    
                                                                 
 max_pooling2d_10 (MaxPooli  (None, 9, 9, 256)         0         
 ng2D)                                                           
                                                                 
 flatten_2 (Flatten)         (None, 20736)             0         
                                                                 
 dense_6 (Dense)             (None, 128)               2654336   
                                                                 
 dropout_4 (Dropout)         (None, 128)               0         
                                                                 
 dense_7 (Dense)             (None, 128)               16512     
                                                                 
 dropout_5 (Dropout)         (None, 128)               0         
                                                                 
 dense_8 (Dense)             (None, 128)               16512     
                                                                 
 dropout_6 (Dropout)         (None, 128)               0         
                                                                 
 dense_9 (Dense)             (None, 5)                 645       
                                                                 
=================================================================
Total params: 3095749 (11.81 MB)
Trainable params: 3095749 (11.81 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 30s 73ms/step - loss: 1.8653 - accuracy: 0.2042 - f1_score: 0.1876 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.7054 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00 - lr: 5.0000e-04
Epoch 2/25
379/379 [==============================] - 22s 57ms/step - loss: 1.6520 - accuracy: 0.2422 - f1_score: 0.2173 - precision_5: 0.4932 - recall_5: 0.0060 - val_loss: 1.5288 - val_accuracy: 0.4217 - val_f1_score: 0.3520 - val_precision_5: 0.8571 - val_recall_5: 0.0045 - lr: 5.0000e-04
Epoch 3/25
379/379 [==============================] - 24s 62ms/step - loss: 1.2768 - accuracy: 0.4760 - f1_score: 0.4731 - precision_5: 0.7471 - recall_5: 0.1902 - val_loss: 1.0374 - val_accuracy: 0.5575 - val_f1_score: 0.5061 - val_precision_5: 0.7948 - val_recall_5: 0.3163 - lr: 5.0000e-04
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 1.0582 - accuracy: 0.5549 - f1_score: 0.5573 - precision_5: 0.8106 - recall_5: 0.3195 - val_loss: 0.9192 - val_accuracy: 0.6117 - val_f1_score: 0.5526 - val_precision_5: 0.8403 - val_recall_5: 0.3868 - lr: 5.0000e-04
Epoch 5/25
379/379 [==============================] - 21s 53ms/step - loss: 0.9669 - accuracy: 0.5907 - f1_score: 0.5923 - precision_5: 0.8316 - recall_5: 0.3682 - val_loss: 0.8424 - val_accuracy: 0.6429 - val_f1_score: 0.6136 - val_precision_5: 0.8704 - val_recall_5: 0.4239 - lr: 5.0000e-04
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.9041 - accuracy: 0.6190 - f1_score: 0.6204 - precision_5: 0.8500 - recall_5: 0.3999 - val_loss: 0.8173 - val_accuracy: 0.6511 - val_f1_score: 0.6276 - val_precision_5: 0.9078 - val_recall_5: 0.4165 - lr: 5.0000e-04
Epoch 7/25
379/379 [==============================] - 21s 55ms/step - loss: 0.8496 - accuracy: 0.6460 - f1_score: 0.6452 - precision_5: 0.8587 - recall_5: 0.4343 - val_loss: 0.8013 - val_accuracy: 0.6793 - val_f1_score: 0.6639 - val_precision_5: 0.8571 - val_recall_5: 0.4588 - lr: 5.0000e-04
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 0.8303 - accuracy: 0.6662 - f1_score: 0.6638 - precision_5: 0.8471 - recall_5: 0.4596 - val_loss: 0.7466 - val_accuracy: 0.7231 - val_f1_score: 0.7243 - val_precision_5: 0.8712 - val_recall_5: 0.5271 - lr: 5.0000e-04
Epoch 9/25
379/379 [==============================] - 21s 53ms/step - loss: 0.8180 - accuracy: 0.6773 - f1_score: 0.6732 - precision_5: 0.8301 - recall_5: 0.4915 - val_loss: 0.7545 - val_accuracy: 0.7112 - val_f1_score: 0.7114 - val_precision_5: 0.8530 - val_recall_5: 0.5212 - lr: 5.0000e-04
Epoch 10/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7782 - accuracy: 0.6948 - f1_score: 0.6917 - precision_5: 0.8242 - recall_5: 0.5406 - val_loss: 0.7394 - val_accuracy: 0.7157 - val_f1_score: 0.7201 - val_precision_5: 0.8479 - val_recall_5: 0.5627 - lr: 5.0000e-04
Epoch 11/25
379/379 [==============================] - 21s 53ms/step - loss: 0.7614 - accuracy: 0.7057 - f1_score: 0.7042 - precision_5: 0.8223 - recall_5: 0.5642 - val_loss: 0.7425 - val_accuracy: 0.7216 - val_f1_score: 0.7223 - val_precision_5: 0.8549 - val_recall_5: 0.5338 - lr: 5.0000e-04
Epoch 12/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7357 - accuracy: 0.7186 - f1_score: 0.7177 - precision_5: 0.8247 - recall_5: 0.5887 - val_loss: 0.7099 - val_accuracy: 0.7298 - val_f1_score: 0.7307 - val_precision_5: 0.8257 - val_recall_5: 0.6295 - lr: 5.0000e-04
Epoch 13/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7239 - accuracy: 0.7252 - f1_score: 0.7242 - precision_5: 0.8144 - recall_5: 0.6111 - val_loss: 0.6974 - val_accuracy: 0.7461 - val_f1_score: 0.7494 - val_precision_5: 0.8577 - val_recall_5: 0.5999 - lr: 5.0000e-04
Epoch 14/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7049 - accuracy: 0.7393 - f1_score: 0.7388 - precision_5: 0.8240 - recall_5: 0.6307 - val_loss: 0.6897 - val_accuracy: 0.7572 - val_f1_score: 0.7566 - val_precision_5: 0.8235 - val_recall_5: 0.6548 - lr: 5.0000e-04
Epoch 15/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7069 - accuracy: 0.7388 - f1_score: 0.7382 - precision_5: 0.8128 - recall_5: 0.6357 - val_loss: 0.6768 - val_accuracy: 0.7520 - val_f1_score: 0.7520 - val_precision_5: 0.8142 - val_recall_5: 0.6637 - lr: 5.0000e-04
Epoch 16/25
379/379 [==============================] - 21s 53ms/step - loss: 0.6894 - accuracy: 0.7473 - f1_score: 0.7469 - precision_5: 0.8238 - recall_5: 0.6502 - val_loss: 0.6796 - val_accuracy: 0.7431 - val_f1_score: 0.7477 - val_precision_5: 0.8147 - val_recall_5: 0.6823 - lr: 5.0000e-04
Epoch 17/25
379/379 [==============================] - 21s 53ms/step - loss: 0.6853 - accuracy: 0.7496 - f1_score: 0.7495 - precision_5: 0.8198 - recall_5: 0.6565 - val_loss: 0.6788 - val_accuracy: 0.7565 - val_f1_score: 0.7528 - val_precision_5: 0.8005 - val_recall_5: 0.7001 - lr: 5.0000e-04
Epoch 18/25
379/379 [==============================] - 21s 53ms/step - loss: 0.6769 - accuracy: 0.7560 - f1_score: 0.7556 - precision_5: 0.8203 - recall_5: 0.6680 - val_loss: 0.6846 - val_accuracy: 0.7491 - val_f1_score: 0.7502 - val_precision_5: 0.8137 - val_recall_5: 0.6711 - lr: 5.0000e-04
Epoch 19/25
379/379 [==============================] - 21s 53ms/step - loss: 0.6631 - accuracy: 0.7597 - f1_score: 0.7594 - precision_5: 0.8258 - recall_5: 0.6800 - val_loss: 0.6833 - val_accuracy: 0.7424 - val_f1_score: 0.7420 - val_precision_5: 0.7975 - val_recall_5: 0.6986 - lr: 5.0000e-04
Epoch 20/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6542 - accuracy: 0.7616 - f1_score: 0.7615 - precision_5: 0.8256 - recall_5: 0.6861 - val_loss: 0.6773 - val_accuracy: 0.7528 - val_f1_score: 0.7484 - val_precision_5: 0.8057 - val_recall_5: 0.7082 - lr: 5.0000e-04

The combination of dropout with data augmentation and learning rate scheduler provides the best results with a validation accuracy around 0.88. However adding L2 regularization, though it did not affect to the training stability, reduced considerably the classification performance.

ARCHITECTURE 2

We will test another hand-crafted architecture arbitrarily defined just with experimental purposes:

• Depth of the first convolutional layer (input_units): 32
• Kernel size of the convolutional layers (kernel_size): 3
• Blocks of convolution and max-pooling (n_layers): 2
• Number of convolutional layers followed by a max-pooling layer (n_conv): 2
• Number of dense layers in the classification block (n_connections): 1
• Number of units in a dense layer (n_nodes): 512

Same as in the previous experiment we will perform a full training without EarlyStopping to more clearly see the effects of possible overfitting and whether our model needs regularization. Then we will modify the network with different regularization techniques trying to maximize its performance and reduce the overfitting problem.

def create_scratch_model_2(use_batch_normalization=False, use_dropout=False, data_augmentation=None, regularizer=None):
    inputs = keras.Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3))
    
    # (Adjustable) Data augmentation
    if data_augmentation is not None:
        x = data_augmentation(inputs)

    # Feature extraction block
    # Convolution layer
    input_filters = 32
    if regularizer is not None:
        x = layers.Conv2D(input_filters, (3, 3), kernel_regularizer=regularizer, activation="relu")(inputs)
        x = layers.Conv2D(input_filters, (3, 3), kernel_regularizer=regularizer, activation="relu")(x)
    else:
        x = layers.Conv2D(input_filters, (3, 3), activation="relu")(inputs)
        x = layers.Conv2D(input_filters, (3, 3), activation="relu")(x)

    x = layers.MaxPooling2D(pool_size=(2, 2))(x)

    # More convolution layers
    for i in range(2):  # adding variation of layers.
        for j in range(3): # number of convolutions before a pooling
            if regularizer is not None:
                x = layers.Conv2D(input_filters*(2**i),  # increase depth by power of 2 for each conv layer
                                (3, 3), kernel_regularizer=regularizer, activation="relu")(x)
            else:
                x = layers.Conv2D(input_filters*(2**i),  # increase depth by power of 2 for each conv layer
                                (3, 3), activation="relu")(x)
                
        # (Adjustable) Batch normalization layer
        if use_batch_normalization:
            x = layers.BatchNormalization()(x)
        x = layers.MaxPooling2D(pool_size=(2, 2))(x)

    # Flatten layer
    x = layers.Flatten()(x)

    ##################################################
    # Classification block
    # Fully connected layers
    if regularizer is not None:
        x = layers.Dense(512, kernel_regularizer=regularizer, activation="relu")(x)
    else:
        x = layers.Dense(512, activation="relu")(x)
    
    # (Adjustable) Dropout layer
    if use_dropout:
        x = layers.Dropout(rate=0.5)(x)

    # Output layer
    outputs = layers.Dense(5, activation="softmax")(x)

    # Prepare model
    model = keras.Model(inputs=inputs, outputs=outputs)

    model.compile(optimizer="adam",
        loss="categorical_crossentropy",
        metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
    )

    model.summary()

    return model

def train_model(model, train_dataset, val_dataset, epochs=20, lr=0.001, optimizer=keras.optimizers.Adam, checkpoint=None, lr_scheduler=None, early_stop=None):

  model.compile(optimizer=optimizer(learning_rate=lr),
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
  )

  callbacks = []
  if checkpoint is not None:
    callbacks.append(checkpoint)
  if lr_scheduler is not None:
    callbacks.append(lr_scheduler)
  if early_stop is not None:
    callbacks.append(early_stop)

  history = model.fit(train_dataset, epochs=epochs, validation_data=val_dataset, callbacks=callbacks)
  return history

def execute_pipeline(train_dataset, val_dataset, hyperparameters):

  # Set the seed to ensure reproducibility
  keras.utils.set_random_seed(RANDOM_SEED)

  model = create_scratch_model_2(use_batch_normalization=hyperparameters["bn"], use_dropout=hyperparameters["dropout"], data_augmentation=hyperparameters["augmentation"], regularizer=hyperparameters["regularizer"])
  history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
  plot(history)

  return model, history

Experiments

First we will test every regularization technique individually to see how it affects to the resulting training plot.

# Baseline model
checkpoint_filename = "/cnn2_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": None,
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_28"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_33 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_205 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_206 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_115 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_207 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_208 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_209 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_116 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_210 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_211 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_212 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_117 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_28 (Flatten)        (None, 18496)             0         
                                                                 
 dense_92 (Dense)            (None, 512)               9470464   
                                                                 
 dropout_10 (Dropout)        (None, 512)               0         
                                                                 
 dense_93 (Dense)            (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 36s 89ms/step - loss: 1.2124 - accuracy: 0.4705 - f1_score: 0.4777 - precision_57: 0.7225 - recall_57: 0.2691 - val_loss: 0.7686 - val_accuracy: 0.6667 - val_f1_score: 0.6451 - val_precision_57: 0.7588 - val_recall_57: 0.5627
Epoch 2/25
379/379 [==============================] - 26s 67ms/step - loss: 0.6844 - accuracy: 0.7180 - f1_score: 0.7172 - precision_57: 0.7927 - recall_57: 0.6269 - val_loss: 0.6675 - val_accuracy: 0.7431 - val_f1_score: 0.7402 - val_precision_57: 0.7930 - val_recall_57: 0.6741
Epoch 3/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5526 - accuracy: 0.7729 - f1_score: 0.7726 - precision_57: 0.8246 - recall_57: 0.7127 - val_loss: 0.5846 - val_accuracy: 0.7795 - val_f1_score: 0.7798 - val_precision_57: 0.8251 - val_recall_57: 0.6971
Epoch 4/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4266 - accuracy: 0.8272 - f1_score: 0.8273 - precision_57: 0.8649 - recall_57: 0.7844 - val_loss: 0.6945 - val_accuracy: 0.7736 - val_f1_score: 0.7725 - val_precision_57: 0.8021 - val_recall_57: 0.7342
Epoch 5/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3433 - accuracy: 0.8636 - f1_score: 0.8638 - precision_57: 0.8903 - recall_57: 0.8338 - val_loss: 0.8393 - val_accuracy: 0.7550 - val_f1_score: 0.7564 - val_precision_57: 0.7741 - val_recall_57: 0.7402
Epoch 6/25
379/379 [==============================] - 21s 53ms/step - loss: 0.2471 - accuracy: 0.9055 - f1_score: 0.9058 - precision_57: 0.9207 - recall_57: 0.8887 - val_loss: 0.9849 - val_accuracy: 0.7684 - val_f1_score: 0.7672 - val_precision_57: 0.7800 - val_recall_57: 0.7528
Epoch 7/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1931 - accuracy: 0.9280 - f1_score: 0.9281 - precision_57: 0.9386 - recall_57: 0.9180 - val_loss: 1.0575 - val_accuracy: 0.7647 - val_f1_score: 0.7664 - val_precision_57: 0.7738 - val_recall_57: 0.7617
Epoch 8/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1515 - accuracy: 0.9466 - f1_score: 0.9466 - precision_57: 0.9533 - recall_57: 0.9397 - val_loss: 0.9399 - val_accuracy: 0.7580 - val_f1_score: 0.7598 - val_precision_57: 0.7728 - val_recall_57: 0.7476
Epoch 9/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1184 - accuracy: 0.9594 - f1_score: 0.9595 - precision_57: 0.9628 - recall_57: 0.9547 - val_loss: 1.1404 - val_accuracy: 0.7691 - val_f1_score: 0.7701 - val_precision_57: 0.7742 - val_recall_57: 0.7587
Epoch 10/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0960 - accuracy: 0.9678 - f1_score: 0.9679 - precision_57: 0.9711 - recall_57: 0.9651 - val_loss: 1.1966 - val_accuracy: 0.7632 - val_f1_score: 0.7641 - val_precision_57: 0.7699 - val_recall_57: 0.7528
Epoch 11/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0672 - accuracy: 0.9777 - f1_score: 0.9778 - precision_57: 0.9793 - recall_57: 0.9760 - val_loss: 1.2239 - val_accuracy: 0.7751 - val_f1_score: 0.7775 - val_precision_57: 0.7813 - val_recall_57: 0.7691
Epoch 12/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0787 - accuracy: 0.9753 - f1_score: 0.9754 - precision_57: 0.9771 - recall_57: 0.9735 - val_loss: 1.3868 - val_accuracy: 0.7491 - val_f1_score: 0.7487 - val_precision_57: 0.7562 - val_recall_57: 0.7416
Epoch 13/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0745 - accuracy: 0.9770 - f1_score: 0.9770 - precision_57: 0.9782 - recall_57: 0.9750 - val_loss: 1.2066 - val_accuracy: 0.7580 - val_f1_score: 0.7564 - val_precision_57: 0.7671 - val_recall_57: 0.7506
Epoch 14/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0541 - accuracy: 0.9825 - f1_score: 0.9825 - precision_57: 0.9839 - recall_57: 0.9817 - val_loss: 1.2801 - val_accuracy: 0.7565 - val_f1_score: 0.7566 - val_precision_57: 0.7622 - val_recall_57: 0.7520
Epoch 15/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0518 - accuracy: 0.9832 - f1_score: 0.9832 - precision_57: 0.9841 - recall_57: 0.9822 - val_loss: 1.4229 - val_accuracy: 0.7587 - val_f1_score: 0.7618 - val_precision_57: 0.7643 - val_recall_57: 0.7558
Epoch 16/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0467 - accuracy: 0.9861 - f1_score: 0.9862 - precision_57: 0.9866 - recall_57: 0.9858 - val_loss: 1.3883 - val_accuracy: 0.7558 - val_f1_score: 0.7543 - val_precision_57: 0.7590 - val_recall_57: 0.7506
Epoch 17/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0567 - accuracy: 0.9814 - f1_score: 0.9815 - precision_57: 0.9821 - recall_57: 0.9801 - val_loss: 1.3178 - val_accuracy: 0.7476 - val_f1_score: 0.7470 - val_precision_57: 0.7572 - val_recall_57: 0.7431
Epoch 18/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0347 - accuracy: 0.9905 - f1_score: 0.9905 - precision_57: 0.9911 - recall_57: 0.9904 - val_loss: 1.4589 - val_accuracy: 0.7639 - val_f1_score: 0.7649 - val_precision_57: 0.7707 - val_recall_57: 0.7587
Epoch 19/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0382 - accuracy: 0.9875 - f1_score: 0.9875 - precision_57: 0.9877 - recall_57: 0.9871 - val_loss: 1.4153 - val_accuracy: 0.7558 - val_f1_score: 0.7583 - val_precision_57: 0.7636 - val_recall_57: 0.7528
Epoch 20/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0287 - accuracy: 0.9913 - f1_score: 0.9913 - precision_57: 0.9917 - recall_57: 0.9911 - val_loss: 1.4723 - val_accuracy: 0.7610 - val_f1_score: 0.7632 - val_precision_57: 0.7643 - val_recall_57: 0.7558
Epoch 21/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0527 - accuracy: 0.9843 - f1_score: 0.9843 - precision_57: 0.9849 - recall_57: 0.9837 - val_loss: 1.2278 - val_accuracy: 0.7587 - val_f1_score: 0.7604 - val_precision_57: 0.7660 - val_recall_57: 0.7558
Epoch 22/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0446 - accuracy: 0.9862 - f1_score: 0.9862 - precision_57: 0.9869 - recall_57: 0.9856 - val_loss: 1.3276 - val_accuracy: 0.7558 - val_f1_score: 0.7566 - val_precision_57: 0.7618 - val_recall_57: 0.7454
Epoch 23/25
379/379 [==============================] - 21s 53ms/step - loss: 0.0401 - accuracy: 0.9884 - f1_score: 0.9884 - precision_57: 0.9886 - recall_57: 0.9880 - val_loss: 1.2221 - val_accuracy: 0.7758 - val_f1_score: 0.7765 - val_precision_57: 0.7840 - val_recall_57: 0.7706
Epoch 24/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0232 - accuracy: 0.9930 - f1_score: 0.9930 - precision_57: 0.9932 - recall_57: 0.9929 - val_loss: 1.4478 - val_accuracy: 0.7691 - val_f1_score: 0.7713 - val_precision_57: 0.7731 - val_recall_57: 0.7639
Epoch 25/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0426 - accuracy: 0.9873 - f1_score: 0.9873 - precision_57: 0.9878 - recall_57: 0.9867 - val_loss: 1.2916 - val_accuracy: 0.7713 - val_f1_score: 0.7701 - val_precision_57: 0.7769 - val_recall_57: 0.7654

Same as in the previous approximation there is a problem of early overfitting in the network starting in epoch 3, even earlier. Thus, in the following subsection we will apply regularization to check whether it helps to increase the performance of the network.

Test Regularization

In this section several experiments are carried out modifying the previous base architecture:

• Batch normalization layers can be added after the convolutional layers to make the training more stable, allowing a higher learning rate and increasing in turn the speed of the learning process. It also mitigates the drawback of random weight initialization.
• Dropout layers can also be used on the dense layers to eliminate the co-adaptation between them, making the network more robust to possible large individual weights of specific neurons that could dominate among the rest.
• Data augmentation can help to create new synthetic images increasing the number of different samples seen by the network and making the model more robust and independent of the training set.
• Weights regularization restricts the values of the weights to small numbers making their distribution more regular. We will test the two ways of doing this, the L1 regularization and the L2 regularization.

In the following cells we will test several possible combinations of these four elements.

# Batch normalization model
checkpoint_filename = "/cnn2_reg_lr0.001_bn.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": True,
    "dropout": False,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_29"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_34 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_213 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_214 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_118 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_215 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_216 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_217 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 batch_normalization_8 (Bat  (None, 82, 82, 32)        128       
 chNormalization)                                                
                                                                 
 max_pooling2d_119 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_218 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_219 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_220 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 batch_normalization_9 (Bat  (None, 35, 35, 64)        256       
 chNormalization)                                                
                                                                 
 max_pooling2d_120 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_29 (Flatten)        (None, 18496)             0         
                                                                 
 dense_94 (Dense)            (None, 512)               9470464   
                                                                 
 dense_95 (Dense)            (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603653 (36.64 MB)
Trainable params: 9603461 (36.63 MB)
Non-trainable params: 192 (768.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 24s 56ms/step - loss: 1.1163 - accuracy: 0.6641 - f1_score: 0.6627 - precision_59: 0.7285 - recall_59: 0.5772 - val_loss: 1.1193 - val_accuracy: 0.5917 - val_f1_score: 0.5789 - val_precision_59: 0.6352 - val_recall_59: 0.4291
Epoch 2/25
379/379 [==============================] - 21s 54ms/step - loss: 0.4945 - accuracy: 0.8043 - f1_score: 0.8040 - precision_59: 0.8408 - recall_59: 0.7633 - val_loss: 2.1568 - val_accuracy: 0.5709 - val_f1_score: 0.5588 - val_precision_59: 0.5918 - val_recall_59: 0.5434
Epoch 3/25
379/379 [==============================] - 22s 56ms/step - loss: 0.3039 - accuracy: 0.8825 - f1_score: 0.8826 - precision_59: 0.9003 - recall_59: 0.8619 - val_loss: 0.8957 - val_accuracy: 0.6956 - val_f1_score: 0.6766 - val_precision_59: 0.7256 - val_recall_59: 0.6674
Epoch 4/25
379/379 [==============================] - 21s 55ms/step - loss: 0.1974 - accuracy: 0.9286 - f1_score: 0.9286 - precision_59: 0.9363 - recall_59: 0.9199 - val_loss: 7.3120 - val_accuracy: 0.3623 - val_f1_score: 0.2775 - val_precision_59: 0.3670 - val_recall_59: 0.3615
Epoch 5/25
379/379 [==============================] - 21s 55ms/step - loss: 0.1345 - accuracy: 0.9527 - f1_score: 0.9527 - precision_59: 0.9566 - recall_59: 0.9496 - val_loss: 2.7189 - val_accuracy: 0.6273 - val_f1_score: 0.5919 - val_precision_59: 0.6355 - val_recall_59: 0.6214
Epoch 6/25
379/379 [==============================] - 21s 55ms/step - loss: 0.1658 - accuracy: 0.9472 - f1_score: 0.9472 - precision_59: 0.9513 - recall_59: 0.9442 - val_loss: 1.5857 - val_accuracy: 0.6333 - val_f1_score: 0.6491 - val_precision_59: 0.6438 - val_recall_59: 0.6281
Epoch 7/25
379/379 [==============================] - 21s 55ms/step - loss: 0.0962 - accuracy: 0.9673 - f1_score: 0.9673 - precision_59: 0.9703 - recall_59: 0.9659 - val_loss: 1.0487 - val_accuracy: 0.7713 - val_f1_score: 0.7724 - val_precision_59: 0.7769 - val_recall_59: 0.7654
Epoch 8/25
379/379 [==============================] - 21s 55ms/step - loss: 0.0357 - accuracy: 0.9878 - f1_score: 0.9878 - precision_59: 0.9885 - recall_59: 0.9870 - val_loss: 2.7227 - val_accuracy: 0.6503 - val_f1_score: 0.6342 - val_precision_59: 0.6525 - val_recall_59: 0.6496

# Dropout model
checkpoint_filename = "/cnn2_reg_lr0.001_dropout.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": True,
    "regularizer": None,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_30"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_35 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_221 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_222 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_121 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_223 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_224 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_225 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_122 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_226 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_227 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_228 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_123 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_30 (Flatten)        (None, 18496)             0         
                                                                 
 dense_96 (Dense)            (None, 512)               9470464   
                                                                 
 dropout_11 (Dropout)        (None, 512)               0         
                                                                 
 dense_97 (Dense)            (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 23s 54ms/step - loss: 1.0224 - accuracy: 0.5569 - f1_score: 0.5562 - precision_61: 0.7399 - recall_61: 0.3549 - val_loss: 0.8152 - val_accuracy: 0.6414 - val_f1_score: 0.6173 - val_precision_61: 0.7126 - val_recall_61: 0.5449
Epoch 2/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6684 - accuracy: 0.7196 - f1_score: 0.7189 - precision_61: 0.8000 - recall_61: 0.6233 - val_loss: 0.6499 - val_accuracy: 0.7342 - val_f1_score: 0.7309 - val_precision_61: 0.8145 - val_recall_61: 0.6355
Epoch 3/25
379/379 [==============================] - 21s 55ms/step - loss: 0.5300 - accuracy: 0.7819 - f1_score: 0.7816 - precision_61: 0.8340 - recall_61: 0.7214 - val_loss: 0.5879 - val_accuracy: 0.7721 - val_f1_score: 0.7707 - val_precision_61: 0.8177 - val_recall_61: 0.7290
Epoch 4/25
379/379 [==============================] - 21s 53ms/step - loss: 0.4384 - accuracy: 0.8205 - f1_score: 0.8204 - precision_61: 0.8627 - recall_61: 0.7760 - val_loss: 0.6754 - val_accuracy: 0.7676 - val_f1_score: 0.7684 - val_precision_61: 0.7932 - val_recall_61: 0.7402
Epoch 5/25
379/379 [==============================] - 21s 53ms/step - loss: 0.3421 - accuracy: 0.8667 - f1_score: 0.8668 - precision_61: 0.8945 - recall_61: 0.8369 - val_loss: 0.7549 - val_accuracy: 0.7676 - val_f1_score: 0.7706 - val_precision_61: 0.7836 - val_recall_61: 0.7365
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.2539 - accuracy: 0.9047 - f1_score: 0.9048 - precision_61: 0.9220 - recall_61: 0.8865 - val_loss: 0.6749 - val_accuracy: 0.7988 - val_f1_score: 0.8028 - val_precision_61: 0.8103 - val_recall_61: 0.7862
Epoch 7/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1794 - accuracy: 0.9318 - f1_score: 0.9319 - precision_61: 0.9413 - recall_61: 0.9244 - val_loss: 0.9434 - val_accuracy: 0.7921 - val_f1_score: 0.7955 - val_precision_61: 0.7985 - val_recall_61: 0.7854
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 0.1560 - accuracy: 0.9443 - f1_score: 0.9443 - precision_61: 0.9517 - recall_61: 0.9382 - val_loss: 0.9433 - val_accuracy: 0.7862 - val_f1_score: 0.7895 - val_precision_61: 0.7917 - val_recall_61: 0.7817

# L1 model
checkpoint_filename = "/cnn2_reg_lr0.001_l1.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": keras.regularizers.l1(0.001),
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_31"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_36 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_229 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_230 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_124 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_231 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_232 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_233 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_125 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_234 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_235 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_236 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_126 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_31 (Flatten)        (None, 18496)             0         
                                                                 
 dense_98 (Dense)            (None, 512)               9470464   
                                                                 
 dense_99 (Dense)            (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 24s 56ms/step - loss: 5.1337 - accuracy: 0.1975 - f1_score: 0.1339 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.7968 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 21s 55ms/step - loss: 2.8004 - accuracy: 0.1993 - f1_score: 0.1116 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.8072 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 21s 55ms/step - loss: 2.8006 - accuracy: 0.2011 - f1_score: 0.1030 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.7630 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 21s 55ms/step - loss: 2.8006 - accuracy: 0.1972 - f1_score: 0.0979 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.7949 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 21s 55ms/step - loss: 2.8006 - accuracy: 0.2009 - f1_score: 0.0918 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.8066 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 2.8004 - accuracy: 0.2008 - f1_score: 0.1055 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.8206 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 21s 54ms/step - loss: 2.8004 - accuracy: 0.1998 - f1_score: 0.1051 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.8339 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 21s 55ms/step - loss: 2.8003 - accuracy: 0.1984 - f1_score: 0.0962 - precision_63: 0.0000e+00 - recall_63: 0.0000e+00 - val_loss: 2.8133 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_63: 0.0000e+00 - val_recall_63: 0.0000e+00

# L2 model
checkpoint_filename = "/cnn2_reg_lr0.001_l2.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "dropout": False,
    "regularizer": keras.regularizers.l2(0.001),
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_32"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_37 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_237 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_238 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_127 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_239 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_240 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_241 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_128 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_242 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_243 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_244 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_129 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_32 (Flatten)        (None, 18496)             0         
                                                                 
 dense_100 (Dense)           (None, 512)               9470464   
                                                                 
 dense_101 (Dense)           (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 23s 55ms/step - loss: 1.4776 - accuracy: 0.4174 - f1_score: 0.4154 - precision_65: 0.6862 - recall_65: 0.1437 - val_loss: 1.0300 - val_accuracy: 0.6147 - val_f1_score: 0.6116 - val_precision_65: 0.7241 - val_recall_65: 0.4521
Epoch 2/25
379/379 [==============================] - 21s 55ms/step - loss: 0.9451 - accuracy: 0.6303 - f1_score: 0.6240 - precision_65: 0.7552 - recall_65: 0.4711 - val_loss: 0.8501 - val_accuracy: 0.6689 - val_f1_score: 0.6689 - val_precision_65: 0.8040 - val_recall_65: 0.4811
Epoch 3/25
379/379 [==============================] - 21s 55ms/step - loss: 0.8415 - accuracy: 0.6715 - f1_score: 0.6670 - precision_65: 0.7760 - recall_65: 0.5315 - val_loss: 0.8030 - val_accuracy: 0.7045 - val_f1_score: 0.7044 - val_precision_65: 0.7931 - val_recall_65: 0.5835
Epoch 4/25
379/379 [==============================] - 21s 55ms/step - loss: 0.7840 - accuracy: 0.7049 - f1_score: 0.7033 - precision_65: 0.7931 - recall_65: 0.5974 - val_loss: 0.7996 - val_accuracy: 0.6934 - val_f1_score: 0.6879 - val_precision_65: 0.7901 - val_recall_65: 0.5702
Epoch 5/25
379/379 [==============================] - 21s 54ms/step - loss: 0.7401 - accuracy: 0.7243 - f1_score: 0.7230 - precision_65: 0.8039 - recall_65: 0.6353 - val_loss: 0.7889 - val_accuracy: 0.7164 - val_f1_score: 0.7141 - val_precision_65: 0.7939 - val_recall_65: 0.6006
Epoch 6/25
379/379 [==============================] - 21s 55ms/step - loss: 0.7075 - accuracy: 0.7416 - f1_score: 0.7408 - precision_65: 0.8082 - recall_65: 0.6598 - val_loss: 0.7113 - val_accuracy: 0.7498 - val_f1_score: 0.7534 - val_precision_65: 0.8051 - val_recall_65: 0.6748
Epoch 7/25
379/379 [==============================] - 21s 55ms/step - loss: 0.6885 - accuracy: 0.7567 - f1_score: 0.7561 - precision_65: 0.8171 - recall_65: 0.6804 - val_loss: 0.7102 - val_accuracy: 0.7439 - val_f1_score: 0.7437 - val_precision_65: 0.8091 - val_recall_65: 0.6733
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6479 - accuracy: 0.7788 - f1_score: 0.7783 - precision_65: 0.8291 - recall_65: 0.7135 - val_loss: 0.7238 - val_accuracy: 0.7491 - val_f1_score: 0.7478 - val_precision_65: 0.8059 - val_recall_65: 0.6659
Epoch 9/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6434 - accuracy: 0.7813 - f1_score: 0.7808 - precision_65: 0.8329 - recall_65: 0.7233 - val_loss: 0.7283 - val_accuracy: 0.7550 - val_f1_score: 0.7588 - val_precision_65: 0.8076 - val_recall_65: 0.6978
Epoch 10/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6171 - accuracy: 0.8004 - f1_score: 0.8003 - precision_65: 0.8459 - recall_65: 0.7504 - val_loss: 0.7524 - val_accuracy: 0.7446 - val_f1_score: 0.7432 - val_precision_65: 0.7988 - val_recall_65: 0.6808
Epoch 11/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5867 - accuracy: 0.8159 - f1_score: 0.8159 - precision_65: 0.8574 - recall_65: 0.7711 - val_loss: 0.7507 - val_accuracy: 0.7565 - val_f1_score: 0.7556 - val_precision_65: 0.8017 - val_recall_65: 0.7112
Epoch 12/25
379/379 [==============================] - 21s 54ms/step - loss: 0.5754 - accuracy: 0.8319 - f1_score: 0.8317 - precision_65: 0.8673 - recall_65: 0.7910 - val_loss: 0.8441 - val_accuracy: 0.7327 - val_f1_score: 0.7309 - val_precision_65: 0.7706 - val_recall_65: 0.6860

# Data augmentation model
checkpoint_filename = "/cnn2_reg_lr0.001_da.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": None,
    "dropout": False,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_33"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_38 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 conv2d_245 (Conv2D)         (None, 178, 178, 32)      896       
                                                                 
 conv2d_246 (Conv2D)         (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d_130 (MaxPool  (None, 88, 88, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_247 (Conv2D)         (None, 86, 86, 32)        9248      
                                                                 
 conv2d_248 (Conv2D)         (None, 84, 84, 32)        9248      
                                                                 
 conv2d_249 (Conv2D)         (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_131 (MaxPool  (None, 41, 41, 32)        0         
 ing2D)                                                          
                                                                 
 conv2d_250 (Conv2D)         (None, 39, 39, 64)        18496     
                                                                 
 conv2d_251 (Conv2D)         (None, 37, 37, 64)        36928     
                                                                 
 conv2d_252 (Conv2D)         (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_132 (MaxPool  (None, 17, 17, 64)        0         
 ing2D)                                                          
                                                                 
 flatten_33 (Flatten)        (None, 18496)             0         
                                                                 
 dense_102 (Dense)           (None, 512)               9470464   
                                                                 
 dense_103 (Dense)           (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 23s 54ms/step - loss: 1.0759 - accuracy: 0.5256 - f1_score: 0.5238 - precision_67: 0.7506 - recall_67: 0.3274 - val_loss: 0.8041 - val_accuracy: 0.6548 - val_f1_score: 0.6359 - val_precision_67: 0.8256 - val_recall_67: 0.4781
Epoch 2/25
379/379 [==============================] - 21s 54ms/step - loss: 0.6726 - accuracy: 0.7114 - f1_score: 0.7112 - precision_67: 0.8069 - recall_67: 0.6077 - val_loss: 0.7065 - val_accuracy: 0.7216 - val_f1_score: 0.7175 - val_precision_67: 0.7795 - val_recall_67: 0.6377
Epoch 3/25
379/379 [==============================] - 21s 55ms/step - loss: 0.4971 - accuracy: 0.7956 - f1_score: 0.7956 - precision_67: 0.8428 - recall_67: 0.7382 - val_loss: 0.6807 - val_accuracy: 0.7691 - val_f1_score: 0.7704 - val_precision_67: 0.8018 - val_recall_67: 0.7357
Epoch 4/25
379/379 [==============================] - 22s 55ms/step - loss: 0.3516 - accuracy: 0.8580 - f1_score: 0.8583 - precision_67: 0.8872 - recall_67: 0.8291 - val_loss: 0.8402 - val_accuracy: 0.7543 - val_f1_score: 0.7567 - val_precision_67: 0.7831 - val_recall_67: 0.7372
Epoch 5/25
379/379 [==============================] - 21s 55ms/step - loss: 0.2333 - accuracy: 0.9070 - f1_score: 0.9072 - precision_67: 0.9259 - recall_67: 0.8940 - val_loss: 1.0411 - val_accuracy: 0.7647 - val_f1_score: 0.7689 - val_precision_67: 0.7689 - val_recall_67: 0.7461
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.1595 - accuracy: 0.9406 - f1_score: 0.9407 - precision_67: 0.9492 - recall_67: 0.9329 - val_loss: 1.2893 - val_accuracy: 0.7298 - val_f1_score: 0.7333 - val_precision_67: 0.7349 - val_recall_67: 0.7246
Epoch 7/25
379/379 [==============================] - 21s 53ms/step - loss: 0.1112 - accuracy: 0.9622 - f1_score: 0.9623 - precision_67: 0.9650 - recall_67: 0.9579 - val_loss: 1.2975 - val_accuracy: 0.7543 - val_f1_score: 0.7584 - val_precision_67: 0.7587 - val_recall_67: 0.7468
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 0.0829 - accuracy: 0.9726 - f1_score: 0.9726 - precision_67: 0.9745 - recall_67: 0.9699 - val_loss: 1.4628 - val_accuracy: 0.7402 - val_f1_score: 0.7426 - val_precision_67: 0.7474 - val_recall_67: 0.7335

# Dropout augmented L2 model
checkpoint_filename = "/cnn2_reg_lr0.001_dropout_aug.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": keras.regularizers.l2(0.001),
    "dropout": True,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d (Conv2D)             (None, 178, 178, 32)      896       
                                                                 
 conv2d_1 (Conv2D)           (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d (MaxPooling2  (None, 88, 88, 32)        0         
 D)                                                              
                                                                 
 conv2d_2 (Conv2D)           (None, 86, 86, 32)        9248      
                                                                 
 conv2d_3 (Conv2D)           (None, 84, 84, 32)        9248      
                                                                 
 conv2d_4 (Conv2D)           (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 41, 41, 32)        0         
 g2D)                                                            
                                                                 
 conv2d_5 (Conv2D)           (None, 39, 39, 64)        18496     
                                                                 
 conv2d_6 (Conv2D)           (None, 37, 37, 64)        36928     
                                                                 
 conv2d_7 (Conv2D)           (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 17, 17, 64)        0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 18496)             0         
                                                                 
 dense (Dense)               (None, 512)               9470464   
                                                                 
 dropout (Dropout)           (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25

2024-02-24 20:19:18.465209: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-24 20:19:21.879486: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f78500393e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-24 20:19:21.879523: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-24 20:19:21.891985: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1708802361.998434    2629 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 61s 118ms/step - loss: 1.7337 - accuracy: 0.2015 - f1_score: 0.1617 - precision_1: 0.3226 - recall_1: 8.2461e-04 - val_loss: 1.6411 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 34s 89ms/step - loss: 1.6296 - accuracy: 0.1963 - f1_score: 0.1317 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6212 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 34s 88ms/step - loss: 1.6175 - accuracy: 0.1969 - f1_score: 0.1253 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6143 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 32s 82ms/step - loss: 1.6128 - accuracy: 0.1977 - f1_score: 0.1158 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6114 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 32s 82ms/step - loss: 1.6109 - accuracy: 0.1994 - f1_score: 0.1071 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6102 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 27s 69ms/step - loss: 1.6100 - accuracy: 0.2005 - f1_score: 0.1195 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6097 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 22s 56ms/step - loss: 1.6098 - accuracy: 0.2003 - f1_score: 0.1127 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6095 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 22s 56ms/step - loss: 1.6097 - accuracy: 0.1960 - f1_score: 0.1197 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 9/25
379/379 [==============================] - 21s 55ms/step - loss: 1.6095 - accuracy: 0.1995 - f1_score: 0.1153 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 10/25
379/379 [==============================] - 21s 55ms/step - loss: 1.6096 - accuracy: 0.1953 - f1_score: 0.1038 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 11/25
379/379 [==============================] - 21s 55ms/step - loss: 1.6095 - accuracy: 0.2005 - f1_score: 0.1056 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 12/25
379/379 [==============================] - 32s 82ms/step - loss: 1.6095 - accuracy: 0.2001 - f1_score: 0.0960 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 13/25
379/379 [==============================] - 33s 86ms/step - loss: 1.6095 - accuracy: 0.2011 - f1_score: 0.1103 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00
Epoch 14/25
379/379 [==============================] - 30s 77ms/step - loss: 1.6095 - accuracy: 0.1996 - f1_score: 0.1118 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_1: 0.0000e+00 - val_recall_1: 0.0000e+00

# Dropout augmented model with learning rate scheduler
checkpoint_filename = "/cnn2_reg_lr0.0005_dropout_aug_lrScheduler.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0005,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": None,
    "dropout": True,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 conv2d (Conv2D)             (None, 178, 178, 32)      896       
                                                                 
 conv2d_1 (Conv2D)           (None, 176, 176, 32)      9248      
                                                                 
 max_pooling2d (MaxPooling2  (None, 88, 88, 32)        0         
 D)                                                              
                                                                 
 conv2d_2 (Conv2D)           (None, 86, 86, 32)        9248      
                                                                 
 conv2d_3 (Conv2D)           (None, 84, 84, 32)        9248      
                                                                 
 conv2d_4 (Conv2D)           (None, 82, 82, 32)        9248      
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 41, 41, 32)        0         
 g2D)                                                            
                                                                 
 conv2d_5 (Conv2D)           (None, 39, 39, 64)        18496     
                                                                 
 conv2d_6 (Conv2D)           (None, 37, 37, 64)        36928     
                                                                 
 conv2d_7 (Conv2D)           (None, 35, 35, 64)        36928     
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 17, 17, 64)        0         
 g2D)                                                            
                                                                 
 flatten (Flatten)           (None, 18496)             0         
                                                                 
 dense (Dense)               (None, 512)               9470464   
                                                                 
 dropout (Dropout)           (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 5)                 2565      
                                                                 
=================================================================
Total params: 9603269 (36.63 MB)
Trainable params: 9603269 (36.63 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25

2024-02-28 22:20:16.274744: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-28 22:20:18.679481: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f5c501aa080 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-28 22:20:18.679519: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-28 22:20:18.691778: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1709155218.803094   47295 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 37s 70ms/step - loss: 1.0341 - accuracy: 0.5702 - f1_score: 0.5721 - precision_1: 0.7375 - recall_1: 0.3952 - val_loss: 0.6600 - val_accuracy: 0.7424 - val_f1_score: 0.7446 - val_precision_1: 0.8004 - val_recall_1: 0.6607 - lr: 5.0000e-04
Epoch 2/25
379/379 [==============================] - 21s 55ms/step - loss: 0.6207 - accuracy: 0.7508 - f1_score: 0.7501 - precision_1: 0.8081 - recall_1: 0.6813 - val_loss: 0.5742 - val_accuracy: 0.7721 - val_f1_score: 0.7724 - val_precision_1: 0.8228 - val_recall_1: 0.7171 - lr: 5.0000e-04
Epoch 3/25
379/379 [==============================] - 22s 56ms/step - loss: 0.5013 - accuracy: 0.8016 - f1_score: 0.8012 - precision_1: 0.8467 - recall_1: 0.7512 - val_loss: 0.5403 - val_accuracy: 0.7854 - val_f1_score: 0.7905 - val_precision_1: 0.8258 - val_recall_1: 0.7424 - lr: 5.0000e-04
Epoch 4/25
379/379 [==============================] - 21s 55ms/step - loss: 0.4128 - accuracy: 0.8366 - f1_score: 0.8366 - precision_1: 0.8694 - recall_1: 0.7977 - val_loss: 0.4815 - val_accuracy: 0.8107 - val_f1_score: 0.8114 - val_precision_1: 0.8434 - val_recall_1: 0.7795 - lr: 5.0000e-04
Epoch 5/25
379/379 [==============================] - 21s 54ms/step - loss: 0.3441 - accuracy: 0.8661 - f1_score: 0.8663 - precision_1: 0.8916 - recall_1: 0.8382 - val_loss: 0.5792 - val_accuracy: 0.8099 - val_f1_score: 0.8089 - val_precision_1: 0.8280 - val_recall_1: 0.7936 - lr: 5.0000e-04
Epoch 6/25
379/379 [==============================] - 21s 54ms/step - loss: 0.2738 - accuracy: 0.8936 - f1_score: 0.8938 - precision_1: 0.9127 - recall_1: 0.8736 - val_loss: 0.5662 - val_accuracy: 0.7958 - val_f1_score: 0.7967 - val_precision_1: 0.8213 - val_recall_1: 0.7743 - lr: 5.0000e-04
Epoch 7/25
379/379 [==============================] - 21s 55ms/step - loss: 0.2075 - accuracy: 0.9240 - f1_score: 0.9240 - precision_1: 0.9364 - recall_1: 0.9115 - val_loss: 0.6393 - val_accuracy: 0.8107 - val_f1_score: 0.8112 - val_precision_1: 0.8242 - val_recall_1: 0.8003 - lr: 5.0000e-04
Epoch 8/25
379/379 [==============================] - 21s 54ms/step - loss: 0.1646 - accuracy: 0.9416 - f1_score: 0.9417 - precision_1: 0.9477 - recall_1: 0.9329 - val_loss: 0.5922 - val_accuracy: 0.8166 - val_f1_score: 0.8192 - val_precision_1: 0.8259 - val_recall_1: 0.8062 - lr: 5.0000e-04
Epoch 9/25
379/379 [==============================] - 21s 54ms/step - loss: 0.1402 - accuracy: 0.9484 - f1_score: 0.9485 - precision_1: 0.9552 - recall_1: 0.9425 - val_loss: 0.7740 - val_accuracy: 0.7869 - val_f1_score: 0.7844 - val_precision_1: 0.8025 - val_recall_1: 0.7780 - lr: 5.0000e-04

In this approach we encountered the same problems as with the first architecture, so refer to it for further explanations and discussion. However, in general the results of the different experiments are much lower highlighting the importance of the initial hyperparameter tuning process for finding the architecture most suitable for our problem.

Again, the model with dropout, data augmentation and the learning rate scheduler achieved to maximize the performance of the architecture but only reaching a validation accuracy of ~0.8.

Complex architectures

To end with this section let's test more complex architectures (without pre-training): InceptionResNetV2, VGG16, VGG19, Inception, Xception and Residual

# Functions to define architectures
def create_complex_model(architecture, dropout=False, data_augmentation=None):  
  inputs = keras.Input(shape=(180, 180, 3))

  if data_augmentation is not None:
    x = data_augmentation(inputs)
    x = architecture(include_top=False, weights=None, input_tensor=None, input_shape=(180,180,3), pooling="max")(x)
  else:
    x = architecture(include_top=False, weights=None, input_tensor=None, input_shape=(180,180,3), pooling="max")(inputs)

  x = layers.Flatten()(x)
  
  # Replace the whole classificator module since they are different problems
  x = layers.Dense(256, activation="relu")(x)
  if (dropout):
    x = layers.Dropout(0.5)(x)
  outputs = layers.Dense(5, activation="softmax")(x)

  model = keras.Model(inputs, outputs)

  model.summary()
  
  return model

def train_model(model, train_dataset, val_dataset, epochs=20, lr=0.001, optimizer=keras.optimizers.Adam, checkpoint=None, lr_scheduler=None, early_stop=None):

  model.compile(optimizer=optimizer(learning_rate=lr),
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
  )

  callbacks = []
  if checkpoint is not None:
    callbacks.append(checkpoint)
  if lr_scheduler is not None:
    callbacks.append(lr_scheduler)
  if early_stop is not None:
    callbacks.append(early_stop)

  history = model.fit(train_dataset, epochs=epochs, validation_data=val_dataset, callbacks=callbacks)
  return history

def execute_pipeline_complex(train_dataset, val_dataset, hyperparameters):

  # Set the seed to ensure reproducibility
  keras.utils.set_random_seed(RANDOM_SEED)

  # Create and train model
  model = create_complex_model(hyperparameters["architecture"], hyperparameters["dropout"], hyperparameters["data_augmentation"])
  history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
  plot(history)

  return model, history

# InceptionResNetV2
checkpoint_filename = "/InceptionResNetV2_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.InceptionResNetV2,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_2 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 180, 180, 3)       0         
                                                                 
 inception_resnet_v2 (Funct  (None, 1536)              54336736  
 ional)                                                          
                                                                 
 flatten_1 (Flatten)         (None, 1536)              0         
                                                                 
 dense_4 (Dense)             (None, 256)               393472    
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_5 (Dense)             (None, 5)                 1285      
                                                                 
=================================================================
Total params: 54731493 (208.78 MB)
Trainable params: 54670949 (208.55 MB)
Non-trainable params: 60544 (236.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 308s 594ms/step - loss: 1.3215 - accuracy: 0.4878 - f1_score: 0.4857 - precision_2: 0.6372 - recall_2: 0.2583 - val_loss: 2.5523 - val_accuracy: 0.2502 - val_f1_score: 0.1505 - val_precision_2: 0.2555 - val_recall_2: 0.2316
Epoch 2/25
379/379 [==============================] - 199s 524ms/step - loss: 0.8618 - accuracy: 0.6637 - f1_score: 0.6638 - precision_2: 0.7669 - recall_2: 0.5183 - val_loss: 0.9293 - val_accuracy: 0.6644 - val_f1_score: 0.6464 - val_precision_2: 0.7131 - val_recall_2: 0.5961
Epoch 3/25
379/379 [==============================] - 186s 488ms/step - loss: 0.7417 - accuracy: 0.7125 - f1_score: 0.7124 - precision_2: 0.8186 - recall_2: 0.5892 - val_loss: 0.8434 - val_accuracy: 0.6429 - val_f1_score: 0.6089 - val_precision_2: 0.7014 - val_recall_2: 0.6050
Epoch 4/25
379/379 [==============================] - 112s 295ms/step - loss: 0.6305 - accuracy: 0.7496 - f1_score: 0.7493 - precision_2: 0.8319 - recall_2: 0.6540 - val_loss: 3.0858 - val_accuracy: 0.4373 - val_f1_score: 0.4507 - val_precision_2: 0.4376 - val_recall_2: 0.3541
Epoch 5/25
379/379 [==============================] - 113s 296ms/step - loss: 0.6459 - accuracy: 0.7528 - f1_score: 0.7527 - precision_2: 0.8332 - recall_2: 0.6497 - val_loss: 0.4986 - val_accuracy: 0.8018 - val_f1_score: 0.7968 - val_precision_2: 0.8352 - val_recall_2: 0.7565
Epoch 6/25
379/379 [==============================] - 112s 295ms/step - loss: 0.4747 - accuracy: 0.8238 - f1_score: 0.8239 - precision_2: 0.8693 - recall_2: 0.7664 - val_loss: 0.8058 - val_accuracy: 0.7120 - val_f1_score: 0.6962 - val_precision_2: 0.7886 - val_recall_2: 0.6592
Epoch 7/25
379/379 [==============================] - 112s 295ms/step - loss: 0.4384 - accuracy: 0.8402 - f1_score: 0.8402 - precision_2: 0.8852 - recall_2: 0.7904 - val_loss: 0.5923 - val_accuracy: 0.8040 - val_f1_score: 0.8014 - val_precision_2: 0.8311 - val_recall_2: 0.7817
Epoch 8/25
379/379 [==============================] - 114s 299ms/step - loss: 0.4020 - accuracy: 0.8513 - f1_score: 0.8514 - precision_2: 0.8911 - recall_2: 0.8087 - val_loss: 0.5859 - val_accuracy: 0.8048 - val_f1_score: 0.7995 - val_precision_2: 0.8260 - val_recall_2: 0.7892
Epoch 9/25
379/379 [==============================] - 113s 295ms/step - loss: 0.3796 - accuracy: 0.8623 - f1_score: 0.8624 - precision_2: 0.8979 - recall_2: 0.8216 - val_loss: 0.2929 - val_accuracy: 0.8909 - val_f1_score: 0.8921 - val_precision_2: 0.9110 - val_recall_2: 0.8738
Epoch 10/25
379/379 [==============================] - 114s 300ms/step - loss: 0.4838 - accuracy: 0.8285 - f1_score: 0.8286 - precision_2: 0.8782 - recall_2: 0.7708 - val_loss: 0.4345 - val_accuracy: 0.8382 - val_f1_score: 0.8350 - val_precision_2: 0.8680 - val_recall_2: 0.8107
Epoch 11/25
379/379 [==============================] - 114s 299ms/step - loss: 0.3635 - accuracy: 0.8680 - f1_score: 0.8680 - precision_2: 0.9016 - recall_2: 0.8284 - val_loss: 0.3333 - val_accuracy: 0.8671 - val_f1_score: 0.8686 - val_precision_2: 0.8952 - val_recall_2: 0.8307
Epoch 12/25
379/379 [==============================] - 112s 295ms/step - loss: 0.3257 - accuracy: 0.8802 - f1_score: 0.8802 - precision_2: 0.9090 - recall_2: 0.8460 - val_loss: 0.3004 - val_accuracy: 0.8864 - val_f1_score: 0.8870 - val_precision_2: 0.9094 - val_recall_2: 0.8716
Epoch 13/25
379/379 [==============================] - 113s 296ms/step - loss: 0.3120 - accuracy: 0.8890 - f1_score: 0.8891 - precision_2: 0.9161 - recall_2: 0.8574 - val_loss: 0.4877 - val_accuracy: 0.8486 - val_f1_score: 0.8501 - val_precision_2: 0.8649 - val_recall_2: 0.8315
Epoch 14/25
379/379 [==============================] - 114s 298ms/step - loss: 0.3356 - accuracy: 0.8850 - f1_score: 0.8851 - precision_2: 0.9110 - recall_2: 0.8521 - val_loss: 0.4260 - val_accuracy: 0.8263 - val_f1_score: 0.8284 - val_precision_2: 0.8558 - val_recall_2: 0.8107

# VGG16
checkpoint_filename = "/vgg16_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.vgg16.VGG16,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None 
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_6"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_14 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_5 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg16 (Functional)          (None, 512)               14714688  
                                                                 
 flatten_6 (Flatten)         (None, 512)               0         
                                                                 
 dense_12 (Dense)            (None, 256)               131328    
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_13 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 14847301 (56.64 MB)
Trainable params: 14847301 (56.64 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 131s 338ms/step - loss: 1.6149 - accuracy: 0.1934 - f1_score: 0.1656 - precision_4: 0.2581 - recall_4: 6.5968e-04 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 129s 338ms/step - loss: 1.6096 - accuracy: 0.1922 - f1_score: 0.1402 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 128s 336ms/step - loss: 1.6096 - accuracy: 0.2004 - f1_score: 0.1189 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 128s 336ms/step - loss: 1.6096 - accuracy: 0.1947 - f1_score: 0.1125 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 128s 337ms/step - loss: 1.6095 - accuracy: 0.2011 - f1_score: 0.1237 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 128s 337ms/step - loss: 1.6096 - accuracy: 0.2017 - f1_score: 0.1193 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 128s 335ms/step - loss: 1.6095 - accuracy: 0.1971 - f1_score: 0.1108 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 128s 337ms/step - loss: 1.6095 - accuracy: 0.2022 - f1_score: 0.1288 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 9/25
379/379 [==============================] - 129s 338ms/step - loss: 1.6096 - accuracy: 0.2011 - f1_score: 0.1198 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 10/25
379/379 [==============================] - 128s 336ms/step - loss: 1.6096 - accuracy: 0.1968 - f1_score: 0.1036 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 11/25
379/379 [==============================] - 128s 335ms/step - loss: 1.6095 - accuracy: 0.1970 - f1_score: 0.1003 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 12/25
379/379 [==============================] - 128s 336ms/step - loss: 1.6095 - accuracy: 0.1987 - f1_score: 0.1023 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 13/25
379/379 [==============================] - 129s 338ms/step - loss: 1.6095 - accuracy: 0.1972 - f1_score: 0.1001 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00
Epoch 14/25
379/379 [==============================] - 129s 338ms/step - loss: 1.6095 - accuracy: 0.2001 - f1_score: 0.1069 - precision_4: 0.0000e+00 - recall_4: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_4: 0.0000e+00 - val_recall_4: 0.0000e+00

# VGG19
checkpoint_filename = "/vgg19_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.vgg19.VGG19,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_7"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_16 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_6 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg19 (Functional)          (None, 512)               20024384  
                                                                 
 flatten_7 (Flatten)         (None, 512)               0         
                                                                 
 dense_14 (Dense)            (None, 256)               131328    
                                                                 
 dropout_6 (Dropout)         (None, 256)               0         
                                                                 
 dense_15 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 20156997 (76.89 MB)
Trainable params: 20156997 (76.89 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 127s 325ms/step - loss: 1.6104 - accuracy: 0.1953 - f1_score: 0.1556 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 125s 327ms/step - loss: 1.6098 - accuracy: 0.1969 - f1_score: 0.1361 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 3/25
379/379 [==============================] - 125s 328ms/step - loss: 1.6096 - accuracy: 0.1976 - f1_score: 0.1215 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 4/25
379/379 [==============================] - 124s 324ms/step - loss: 1.6096 - accuracy: 0.2005 - f1_score: 0.1252 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 5/25
379/379 [==============================] - 124s 325ms/step - loss: 1.6096 - accuracy: 0.2000 - f1_score: 0.1307 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 6/25
379/379 [==============================] - 125s 327ms/step - loss: 1.6096 - accuracy: 0.1976 - f1_score: 0.1166 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 7/25
379/379 [==============================] - 124s 325ms/step - loss: 1.6096 - accuracy: 0.2025 - f1_score: 0.1151 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 8/25
379/379 [==============================] - 124s 326ms/step - loss: 1.6095 - accuracy: 0.1998 - f1_score: 0.1150 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 9/25
379/379 [==============================] - 124s 325ms/step - loss: 1.6095 - accuracy: 0.2005 - f1_score: 0.1132 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 10/25
379/379 [==============================] - 124s 324ms/step - loss: 1.6095 - accuracy: 0.1972 - f1_score: 0.1032 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 11/25
379/379 [==============================] - 124s 325ms/step - loss: 1.6095 - accuracy: 0.1992 - f1_score: 0.1080 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 12/25
379/379 [==============================] - 125s 327ms/step - loss: 1.6095 - accuracy: 0.2003 - f1_score: 0.1072 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 13/25
379/379 [==============================] - 124s 326ms/step - loss: 1.6095 - accuracy: 0.1978 - f1_score: 0.1115 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 14/25
379/379 [==============================] - 124s 325ms/step - loss: 1.6095 - accuracy: 0.1985 - f1_score: 0.1030 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 15/25
379/379 [==============================] - 124s 327ms/step - loss: 1.6095 - accuracy: 0.1997 - f1_score: 0.0994 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 16/25
379/379 [==============================] - 123s 324ms/step - loss: 1.6095 - accuracy: 0.2030 - f1_score: 0.1043 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6094 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00
Epoch 17/25
379/379 [==============================] - 124s 326ms/step - loss: 1.6095 - accuracy: 0.1997 - f1_score: 0.1129 - precision_5: 0.0000e+00 - recall_5: 0.0000e+00 - val_loss: 1.6093 - val_accuracy: 0.2004 - val_f1_score: 0.0668 - val_precision_5: 0.0000e+00 - val_recall_5: 0.0000e+00

# Inception
checkpoint_filename = "/inception_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.InceptionV3,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_8"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_18 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_7 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_v3 (Functional)   (None, 2048)              21802784  
                                                                 
 flatten_8 (Flatten)         (None, 2048)              0         
                                                                 
 dense_16 (Dense)            (None, 256)               524544    
                                                                 
 dropout_7 (Dropout)         (None, 256)               0         
                                                                 
 dense_17 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 22328613 (85.18 MB)
Trainable params: 22294181 (85.05 MB)
Non-trainable params: 34432 (134.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 139s 273ms/step - loss: 1.6785 - accuracy: 0.3444 - f1_score: 0.3379 - precision_6: 0.4959 - recall_6: 0.0697 - val_loss: 1.9047 - val_accuracy: 0.2613 - val_f1_score: 0.1867 - val_precision_6: 0.4524 - val_recall_6: 0.0141
Epoch 2/25
379/379 [==============================] - 83s 216ms/step - loss: 1.1364 - accuracy: 0.5371 - f1_score: 0.5364 - precision_6: 0.6452 - recall_6: 0.2938 - val_loss: 1.0558 - val_accuracy: 0.6169 - val_f1_score: 0.5907 - val_precision_6: 0.6607 - val_recall_6: 0.4959
Epoch 3/25
379/379 [==============================] - 83s 216ms/step - loss: 0.8692 - accuracy: 0.6854 - f1_score: 0.6854 - precision_6: 0.7781 - recall_6: 0.5347 - val_loss: 1.3284 - val_accuracy: 0.5382 - val_f1_score: 0.5333 - val_precision_6: 0.6043 - val_recall_6: 0.3957
Epoch 4/25
379/379 [==============================] - 83s 217ms/step - loss: 0.7526 - accuracy: 0.7267 - f1_score: 0.7270 - precision_6: 0.8119 - recall_6: 0.6091 - val_loss: 0.8732 - val_accuracy: 0.6897 - val_f1_score: 0.6702 - val_precision_6: 0.7367 - val_recall_6: 0.6251
Epoch 5/25
379/379 [==============================] - 83s 217ms/step - loss: 0.7771 - accuracy: 0.7044 - f1_score: 0.7037 - precision_6: 0.8023 - recall_6: 0.5821 - val_loss: 1.6268 - val_accuracy: 0.2264 - val_f1_score: 0.1242 - val_precision_6: 0.4889 - val_recall_6: 0.0163
Epoch 6/25
379/379 [==============================] - 20538s 54s/step - loss: 0.7987 - accuracy: 0.6937 - f1_score: 0.6937 - precision_6: 0.8012 - recall_6: 0.5624 - val_loss: 234.2844 - val_accuracy: 0.2561 - val_f1_score: 0.1681 - val_precision_6: 0.2549 - val_recall_6: 0.2517
Epoch 7/25
379/379 [==============================] - 84s 219ms/step - loss: 0.7996 - accuracy: 0.6900 - f1_score: 0.6901 - precision_6: 0.8007 - recall_6: 0.5516 - val_loss: 0.9325 - val_accuracy: 0.6503 - val_f1_score: 0.6444 - val_precision_6: 0.7111 - val_recall_6: 0.5902
Epoch 8/25
379/379 [==============================] - 83s 217ms/step - loss: 0.7210 - accuracy: 0.7141 - f1_score: 0.7140 - precision_6: 0.8099 - recall_6: 0.5981 - val_loss: 0.7041 - val_accuracy: 0.7431 - val_f1_score: 0.7409 - val_precision_6: 0.7893 - val_recall_6: 0.6815
Epoch 9/25
379/379 [==============================] - 83s 217ms/step - loss: 0.7997 - accuracy: 0.6826 - f1_score: 0.6841 - precision_6: 0.7993 - recall_6: 0.5447 - val_loss: 1.3157 - val_accuracy: 0.4744 - val_f1_score: 0.4503 - val_precision_6: 0.6050 - val_recall_6: 0.3059
Epoch 10/25
379/379 [==============================] - 85s 222ms/step - loss: 0.6766 - accuracy: 0.7258 - f1_score: 0.7267 - precision_6: 0.8168 - recall_6: 0.6203 - val_loss: 1.5831 - val_accuracy: 0.2346 - val_f1_score: 0.1300 - val_precision_6: 0.7500 - val_recall_6: 0.0290
Epoch 11/25
379/379 [==============================] - 84s 220ms/step - loss: 0.6228 - accuracy: 0.7584 - f1_score: 0.7600 - precision_6: 0.8367 - recall_6: 0.6696 - val_loss: 1.4231 - val_accuracy: 0.4736 - val_f1_score: 0.4574 - val_precision_6: 0.5197 - val_recall_6: 0.3630
Epoch 12/25
379/379 [==============================] - 84s 218ms/step - loss: 0.6549 - accuracy: 0.7426 - f1_score: 0.7441 - precision_6: 0.8243 - recall_6: 0.6450 - val_loss: 1.3710 - val_accuracy: 0.4751 - val_f1_score: 0.4540 - val_precision_6: 0.5469 - val_recall_6: 0.4284
Epoch 13/25
379/379 [==============================] - 84s 219ms/step - loss: 0.8610 - accuracy: 0.6484 - f1_score: 0.6478 - precision_6: 0.7922 - recall_6: 0.4951 - val_loss: 1.5819 - val_accuracy: 0.4269 - val_f1_score: 0.4403 - val_precision_6: 0.5115 - val_recall_6: 0.2977

# Inception with lower learning rate
checkpoint_filename = "/inception_lr0.0001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0001,
    "optimizer": optimizers.Adam,

    "architecture": applications.InceptionV3,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_12"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_26 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_11 (Sequential)  (None, 180, 180, 3)       0         
                                                                 
 inception_v3 (Functional)   (None, 2048)              21802784  
                                                                 
 flatten_12 (Flatten)        (None, 2048)              0         
                                                                 
 dense_24 (Dense)            (None, 256)               524544    
                                                                 
 dropout_11 (Dropout)        (None, 256)               0         
                                                                 
 dense_25 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 22328613 (85.18 MB)
Trainable params: 22294181 (85.05 MB)
Non-trainable params: 34432 (134.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 106s 229ms/step - loss: 1.5145 - accuracy: 0.3892 - f1_score: 0.3887 - precision_10: 0.5076 - recall_10: 0.1370 - val_loss: 2.6428 - val_accuracy: 0.2264 - val_f1_score: 0.1252 - val_precision_10: 0.2526 - val_recall_10: 0.1997
Epoch 2/25
379/379 [==============================] - 88s 230ms/step - loss: 1.1955 - accuracy: 0.5189 - f1_score: 0.5183 - precision_10: 0.6525 - recall_10: 0.3043 - val_loss: 1.2658 - val_accuracy: 0.5345 - val_f1_score: 0.5277 - val_precision_10: 0.5857 - val_recall_10: 0.3653
Epoch 3/25
379/379 [==============================] - 59s 154ms/step - loss: 1.0334 - accuracy: 0.5968 - f1_score: 0.5957 - precision_10: 0.7052 - recall_10: 0.4332 - val_loss: 1.9138 - val_accuracy: 0.4759 - val_f1_score: 0.4607 - val_precision_10: 0.5040 - val_recall_10: 0.3697
Epoch 4/25
379/379 [==============================] - 61s 159ms/step - loss: 0.9238 - accuracy: 0.6449 - f1_score: 0.6445 - precision_10: 0.7326 - recall_10: 0.5092 - val_loss: 1.3399 - val_accuracy: 0.5561 - val_f1_score: 0.5613 - val_precision_10: 0.6083 - val_recall_10: 0.4670
Epoch 5/25
379/379 [==============================] - 52s 137ms/step - loss: 0.8620 - accuracy: 0.6763 - f1_score: 0.6761 - precision_10: 0.7564 - recall_10: 0.5541 - val_loss: 1.0742 - val_accuracy: 0.6392 - val_f1_score: 0.6417 - val_precision_10: 0.7098 - val_recall_10: 0.5538
Epoch 6/25
379/379 [==============================] - 85s 222ms/step - loss: 0.7838 - accuracy: 0.7011 - f1_score: 0.7005 - precision_10: 0.7764 - recall_10: 0.5970 - val_loss: 0.8557 - val_accuracy: 0.6281 - val_f1_score: 0.6165 - val_precision_10: 0.7333 - val_recall_10: 0.5553
Epoch 7/25
379/379 [==============================] - 57s 148ms/step - loss: 0.7380 - accuracy: 0.7299 - f1_score: 0.7293 - precision_10: 0.7999 - recall_10: 0.6365 - val_loss: 2.0859 - val_accuracy: 0.6652 - val_f1_score: 0.6621 - val_precision_10: 0.6915 - val_recall_10: 0.5791
Epoch 8/25
379/379 [==============================] - 52s 136ms/step - loss: 0.6636 - accuracy: 0.7473 - f1_score: 0.7470 - precision_10: 0.8065 - recall_10: 0.6723 - val_loss: 0.7846 - val_accuracy: 0.7030 - val_f1_score: 0.6997 - val_precision_10: 0.7680 - val_recall_10: 0.6414
Epoch 9/25
379/379 [==============================] - 53s 137ms/step - loss: 0.6349 - accuracy: 0.7620 - f1_score: 0.7617 - precision_10: 0.8186 - recall_10: 0.6888 - val_loss: 1.0334 - val_accuracy: 0.6607 - val_f1_score: 0.6367 - val_precision_10: 0.7096 - val_recall_10: 0.6095
Epoch 10/25
379/379 [==============================] - 52s 137ms/step - loss: 0.5898 - accuracy: 0.7799 - f1_score: 0.7795 - precision_10: 0.8301 - recall_10: 0.7125 - val_loss: 0.7475 - val_accuracy: 0.7201 - val_f1_score: 0.7157 - val_precision_10: 0.7583 - val_recall_10: 0.6800
Epoch 11/25
379/379 [==============================] - 52s 136ms/step - loss: 0.5650 - accuracy: 0.7950 - f1_score: 0.7946 - precision_10: 0.8432 - recall_10: 0.7314 - val_loss: 0.6421 - val_accuracy: 0.7580 - val_f1_score: 0.7531 - val_precision_10: 0.8210 - val_recall_10: 0.6912
Epoch 12/25
379/379 [==============================] - 52s 135ms/step - loss: 0.5364 - accuracy: 0.8018 - f1_score: 0.8013 - precision_10: 0.8472 - recall_10: 0.7464 - val_loss: 0.7270 - val_accuracy: 0.7416 - val_f1_score: 0.7345 - val_precision_10: 0.7834 - val_recall_10: 0.7008
Epoch 13/25
379/379 [==============================] - 52s 135ms/step - loss: 0.5074 - accuracy: 0.8183 - f1_score: 0.8179 - precision_10: 0.8558 - recall_10: 0.7675 - val_loss: 0.6463 - val_accuracy: 0.7424 - val_f1_score: 0.7422 - val_precision_10: 0.7811 - val_recall_10: 0.6993
Epoch 14/25
379/379 [==============================] - 52s 134ms/step - loss: 0.4783 - accuracy: 0.8244 - f1_score: 0.8243 - precision_10: 0.8655 - recall_10: 0.7766 - val_loss: 0.5986 - val_accuracy: 0.7765 - val_f1_score: 0.7742 - val_precision_10: 0.8229 - val_recall_10: 0.7246
Epoch 15/25
379/379 [==============================] - 52s 135ms/step - loss: 0.4654 - accuracy: 0.8274 - f1_score: 0.8273 - precision_10: 0.8661 - recall_10: 0.7845 - val_loss: 0.9196 - val_accuracy: 0.7105 - val_f1_score: 0.7082 - val_precision_10: 0.7355 - val_recall_10: 0.6793
Epoch 16/25
379/379 [==============================] - 52s 136ms/step - loss: 0.4235 - accuracy: 0.8456 - f1_score: 0.8456 - precision_10: 0.8778 - recall_10: 0.8127 - val_loss: 0.7623 - val_accuracy: 0.7513 - val_f1_score: 0.7591 - val_precision_10: 0.7803 - val_recall_10: 0.7275
Epoch 17/25
379/379 [==============================] - 52s 135ms/step - loss: 0.4206 - accuracy: 0.8429 - f1_score: 0.8429 - precision_10: 0.8794 - recall_10: 0.8041 - val_loss: 0.6612 - val_accuracy: 0.7825 - val_f1_score: 0.7881 - val_precision_10: 0.8120 - val_recall_10: 0.7535
Epoch 18/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3822 - accuracy: 0.8554 - f1_score: 0.8554 - precision_10: 0.8855 - recall_10: 0.8216 - val_loss: 0.6284 - val_accuracy: 0.7854 - val_f1_score: 0.7930 - val_precision_10: 0.8213 - val_recall_10: 0.7506
Epoch 19/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3861 - accuracy: 0.8623 - f1_score: 0.8623 - precision_10: 0.8897 - recall_10: 0.8277 - val_loss: 0.4873 - val_accuracy: 0.8033 - val_f1_score: 0.8039 - val_precision_10: 0.8613 - val_recall_10: 0.7654
Epoch 20/25
379/379 [==============================] - 52s 136ms/step - loss: 0.3653 - accuracy: 0.8652 - f1_score: 0.8652 - precision_10: 0.8929 - recall_10: 0.8352 - val_loss: 0.5521 - val_accuracy: 0.8107 - val_f1_score: 0.8138 - val_precision_10: 0.8333 - val_recall_10: 0.7944
Epoch 21/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3517 - accuracy: 0.8729 - f1_score: 0.8730 - precision_10: 0.8976 - recall_10: 0.8457 - val_loss: 0.6169 - val_accuracy: 0.7825 - val_f1_score: 0.7912 - val_precision_10: 0.8023 - val_recall_10: 0.7624
Epoch 22/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3347 - accuracy: 0.8771 - f1_score: 0.8771 - precision_10: 0.8995 - recall_10: 0.8502 - val_loss: 0.4889 - val_accuracy: 0.8263 - val_f1_score: 0.8287 - val_precision_10: 0.8432 - val_recall_10: 0.8025
Epoch 23/25
379/379 [==============================] - 52s 136ms/step - loss: 0.3093 - accuracy: 0.8870 - f1_score: 0.8872 - precision_10: 0.9077 - recall_10: 0.8622 - val_loss: 0.3898 - val_accuracy: 0.8486 - val_f1_score: 0.8517 - val_precision_10: 0.8621 - val_recall_10: 0.8263
Epoch 24/25
379/379 [==============================] - 52s 135ms/step - loss: 0.2877 - accuracy: 0.8935 - f1_score: 0.8935 - precision_10: 0.9107 - recall_10: 0.8742 - val_loss: 0.4271 - val_accuracy: 0.8434 - val_f1_score: 0.8459 - val_precision_10: 0.8699 - val_recall_10: 0.8189
Epoch 25/25
379/379 [==============================] - 52s 135ms/step - loss: 0.2869 - accuracy: 0.8963 - f1_score: 0.8963 - precision_10: 0.9151 - recall_10: 0.8743 - val_loss: 0.3923 - val_accuracy: 0.8537 - val_f1_score: 0.8557 - val_precision_10: 0.8748 - val_recall_10: 0.8300

# Xception
checkpoint_filename = "/xception_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.Xception,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),

    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 180, 180, 3)       0         
                                                                 
 xception (Functional)       (None, 2048)              20861480  
                                                                 
 flatten (Flatten)           (None, 2048)              0         
                                                                 
 dense (Dense)               (None, 256)               524544    
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 5)                 1285      
                                                                 
=================================================================
Total params: 21387309 (81.59 MB)
Trainable params: 21332781 (81.38 MB)
Non-trainable params: 54528 (213.00 KB)
_________________________________________________________________
Epoch 1/25

2024-02-29 20:46:01.066942: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-29 20:46:05.050554: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f4c0cc72ed0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-29 20:46:05.050589: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-29 20:46:05.064587: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1709235965.168577   39188 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 111s 219ms/step - loss: 1.4722 - accuracy: 0.4049 - f1_score: 0.4020 - precision: 0.5197 - recall: 0.1380 - val_loss: 1.6391 - val_accuracy: 0.2027 - val_f1_score: 0.0689 - val_precision: 0.0000e+00 - val_recall: 0.0000e+00
Epoch 2/25
379/379 [==============================] - 74s 192ms/step - loss: 1.0635 - accuracy: 0.5442 - f1_score: 0.5442 - precision: 0.7080 - recall: 0.3345 - val_loss: 0.9161 - val_accuracy: 0.6221 - val_f1_score: 0.6054 - val_precision: 0.7670 - val_recall: 0.4692
Epoch 3/25
379/379 [==============================] - 74s 193ms/step - loss: 0.8547 - accuracy: 0.6378 - f1_score: 0.6366 - precision: 0.7721 - recall: 0.4740 - val_loss: 0.8612 - val_accuracy: 0.6555 - val_f1_score: 0.6407 - val_precision: 0.7704 - val_recall: 0.5108
Epoch 4/25
379/379 [==============================] - 72s 189ms/step - loss: 0.7427 - accuracy: 0.6971 - f1_score: 0.6982 - precision: 0.8038 - recall: 0.5620 - val_loss: 0.8774 - val_accuracy: 0.7082 - val_f1_score: 0.6858 - val_precision: 0.7964 - val_recall: 0.5865
Epoch 5/25
379/379 [==============================] - 74s 193ms/step - loss: 0.6916 - accuracy: 0.7145 - f1_score: 0.7154 - precision: 0.8171 - recall: 0.5978 - val_loss: 0.6229 - val_accuracy: 0.7424 - val_f1_score: 0.7198 - val_precision: 0.8070 - val_recall: 0.7142
Epoch 6/25
379/379 [==============================] - 73s 190ms/step - loss: 0.6453 - accuracy: 0.7352 - f1_score: 0.7364 - precision: 0.8348 - recall: 0.6302 - val_loss: 0.8344 - val_accuracy: 0.6451 - val_f1_score: 0.5772 - val_precision: 0.7233 - val_recall: 0.6095
Epoch 7/25
379/379 [==============================] - 73s 190ms/step - loss: 0.6055 - accuracy: 0.7633 - f1_score: 0.7639 - precision: 0.8498 - recall: 0.6675 - val_loss: 0.9992 - val_accuracy: 0.6385 - val_f1_score: 0.5879 - val_precision: 0.6685 - val_recall: 0.6258
Epoch 8/25
379/379 [==============================] - 73s 190ms/step - loss: 0.5921 - accuracy: 0.7594 - f1_score: 0.7611 - precision: 0.8515 - recall: 0.6693 - val_loss: 1.1418 - val_accuracy: 0.6763 - val_f1_score: 0.6517 - val_precision: 0.6992 - val_recall: 0.6437
Epoch 9/25
379/379 [==============================] - 73s 190ms/step - loss: 0.5638 - accuracy: 0.7709 - f1_score: 0.7725 - precision: 0.8520 - recall: 0.6873 - val_loss: 0.6439 - val_accuracy: 0.8062 - val_f1_score: 0.8028 - val_precision: 0.8433 - val_recall: 0.7550
Epoch 10/25
379/379 [==============================] - 73s 192ms/step - loss: 0.5375 - accuracy: 0.7846 - f1_score: 0.7859 - precision: 0.8611 - recall: 0.7073 - val_loss: 0.4767 - val_accuracy: 0.8300 - val_f1_score: 0.8302 - val_precision: 0.8907 - val_recall: 0.7684
Epoch 11/25
379/379 [==============================] - 73s 192ms/step - loss: 0.5274 - accuracy: 0.7906 - f1_score: 0.7917 - precision: 0.8702 - recall: 0.7158 - val_loss: 0.4751 - val_accuracy: 0.8033 - val_f1_score: 0.7998 - val_precision: 0.8337 - val_recall: 0.7669
Epoch 12/25
379/379 [==============================] - 74s 193ms/step - loss: 0.5183 - accuracy: 0.7985 - f1_score: 0.8000 - precision: 0.8760 - recall: 0.7293 - val_loss: 0.4254 - val_accuracy: 0.8218 - val_f1_score: 0.8205 - val_precision: 0.8695 - val_recall: 0.7765
Epoch 13/25
379/379 [==============================] - 73s 192ms/step - loss: 0.4849 - accuracy: 0.8149 - f1_score: 0.8161 - precision: 0.8815 - recall: 0.7492 - val_loss: 0.4030 - val_accuracy: 0.8441 - val_f1_score: 0.8463 - val_precision: 0.8818 - val_recall: 0.8085
Epoch 14/25
379/379 [==============================] - 72s 189ms/step - loss: 0.4678 - accuracy: 0.8194 - f1_score: 0.8207 - precision: 0.8848 - recall: 0.7635 - val_loss: 0.4743 - val_accuracy: 0.8471 - val_f1_score: 0.8493 - val_precision: 0.8635 - val_recall: 0.8218
Epoch 15/25
379/379 [==============================] - 73s 190ms/step - loss: 0.4268 - accuracy: 0.8377 - f1_score: 0.8388 - precision: 0.8945 - recall: 0.7848 - val_loss: 0.4328 - val_accuracy: 0.8486 - val_f1_score: 0.8539 - val_precision: 0.8742 - val_recall: 0.8255
Epoch 16/25
379/379 [==============================] - 74s 193ms/step - loss: 0.4058 - accuracy: 0.8456 - f1_score: 0.8467 - precision: 0.8968 - recall: 0.7963 - val_loss: 0.2641 - val_accuracy: 0.8931 - val_f1_score: 0.8947 - val_precision: 0.9224 - val_recall: 0.8738
Epoch 17/25
379/379 [==============================] - 72s 189ms/step - loss: 0.4017 - accuracy: 0.8437 - f1_score: 0.8447 - precision: 0.8957 - recall: 0.7932 - val_loss: 0.3777 - val_accuracy: 0.8627 - val_f1_score: 0.8623 - val_precision: 0.8856 - val_recall: 0.8389
Epoch 18/25
379/379 [==============================] - 73s 192ms/step - loss: 0.4036 - accuracy: 0.8477 - f1_score: 0.8485 - precision: 0.8971 - recall: 0.8027 - val_loss: 0.2494 - val_accuracy: 0.9072 - val_f1_score: 0.9085 - val_precision: 0.9261 - val_recall: 0.8842
Epoch 19/25
379/379 [==============================] - 72s 190ms/step - loss: 0.3964 - accuracy: 0.8576 - f1_score: 0.8583 - precision: 0.9031 - recall: 0.8089 - val_loss: 0.6233 - val_accuracy: 0.8003 - val_f1_score: 0.7917 - val_precision: 0.8453 - val_recall: 0.7624
Epoch 20/25
379/379 [==============================] - 73s 190ms/step - loss: 0.3954 - accuracy: 0.8527 - f1_score: 0.8535 - precision: 0.9021 - recall: 0.8064 - val_loss: 0.3289 - val_accuracy: 0.8931 - val_f1_score: 0.8938 - val_precision: 0.9108 - val_recall: 0.8797
Epoch 21/25
379/379 [==============================] - 72s 190ms/step - loss: 0.3604 - accuracy: 0.8672 - f1_score: 0.8682 - precision: 0.9130 - recall: 0.8232 - val_loss: 0.2602 - val_accuracy: 0.9124 - val_f1_score: 0.9131 - val_precision: 0.9310 - val_recall: 0.9020
Epoch 22/25
379/379 [==============================] - 72s 190ms/step - loss: 0.3497 - accuracy: 0.8716 - f1_score: 0.8729 - precision: 0.9174 - recall: 0.8314 - val_loss: 0.2897 - val_accuracy: 0.8931 - val_f1_score: 0.8945 - val_precision: 0.9252 - val_recall: 0.8537
Epoch 23/25
379/379 [==============================] - 74s 192ms/step - loss: 0.3376 - accuracy: 0.8765 - f1_score: 0.8773 - precision: 0.9181 - recall: 0.8376 - val_loss: 0.1993 - val_accuracy: 0.9235 - val_f1_score: 0.9242 - val_precision: 0.9393 - val_recall: 0.9079
Epoch 24/25
379/379 [==============================] - 72s 189ms/step - loss: 0.3015 - accuracy: 0.8893 - f1_score: 0.8904 - precision: 0.9273 - recall: 0.8568 - val_loss: 0.8701 - val_accuracy: 0.7394 - val_f1_score: 0.7328 - val_precision: 0.7522 - val_recall: 0.7120
Epoch 25/25
379/379 [==============================] - 73s 190ms/step - loss: 0.3188 - accuracy: 0.8884 - f1_score: 0.8892 - precision: 0.9275 - recall: 0.8511 - val_loss: 0.3518 - val_accuracy: 0.8901 - val_f1_score: 0.8918 - val_precision: 0.9075 - val_recall: 0.8812

# Residual
checkpoint_filename = "/residual_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "architecture": applications.ResNet50,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_10"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_22 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_9 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 resnet50 (Functional)       (None, 2048)              23587712  
                                                                 
 flatten_10 (Flatten)        (None, 2048)              0         
                                                                 
 dense_20 (Dense)            (None, 256)               524544    
                                                                 
 dropout_9 (Dropout)         (None, 256)               0         
                                                                 
 dense_21 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 24113541 (91.99 MB)
Trainable params: 24060421 (91.78 MB)
Non-trainable params: 53120 (207.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 110s 214ms/step - loss: 1.9464 - accuracy: 0.2438 - f1_score: 0.2214 - precision_8: 0.3087 - recall_8: 0.0148 - val_loss: 1.6659 - val_accuracy: 0.2094 - val_f1_score: 0.0829 - val_precision_8: 0.6957 - val_recall_8: 0.0119
Epoch 2/25
379/379 [==============================] - 71s 185ms/step - loss: 1.5368 - accuracy: 0.2898 - f1_score: 0.2496 - precision_8: 0.5761 - recall_8: 0.0350 - val_loss: 1.4489 - val_accuracy: 0.3229 - val_f1_score: 0.2952 - val_precision_8: 0.6739 - val_recall_8: 0.0460
Epoch 3/25
379/379 [==============================] - 71s 185ms/step - loss: 1.3797 - accuracy: 0.3650 - f1_score: 0.3677 - precision_8: 0.5204 - recall_8: 0.0915 - val_loss: 1.1437 - val_accuracy: 0.4603 - val_f1_score: 0.4280 - val_precision_8: 0.4479 - val_recall_8: 0.0542
Epoch 4/25
379/379 [==============================] - 82s 214ms/step - loss: 1.2350 - accuracy: 0.4225 - f1_score: 0.4230 - precision_8: 0.5381 - recall_8: 0.1484 - val_loss: 1.2920 - val_accuracy: 0.3630 - val_f1_score: 0.2844 - val_precision_8: 0.4859 - val_recall_8: 0.0898
Epoch 5/25
379/379 [==============================] - 119s 312ms/step - loss: 1.1643 - accuracy: 0.4434 - f1_score: 0.4438 - precision_8: 0.5483 - recall_8: 0.1861 - val_loss: 4.0336 - val_accuracy: 0.1589 - val_f1_score: 0.1033 - val_precision_8: 0.1358 - val_recall_8: 0.0349
Epoch 6/25
379/379 [==============================] - 120s 316ms/step - loss: 1.1906 - accuracy: 0.4422 - f1_score: 0.4424 - precision_8: 0.5801 - recall_8: 0.2007 - val_loss: 1.5236 - val_accuracy: 0.4172 - val_f1_score: 0.4109 - val_precision_8: 0.5437 - val_recall_8: 0.1938
Epoch 7/25
379/379 [==============================] - 120s 316ms/step - loss: 1.2627 - accuracy: 0.4075 - f1_score: 0.4053 - precision_8: 0.5386 - recall_8: 0.1549 - val_loss: 1.2234 - val_accuracy: 0.4484 - val_f1_score: 0.3874 - val_precision_8: 0.5317 - val_recall_8: 0.2799
Epoch 8/25
379/379 [==============================] - 120s 316ms/step - loss: 1.1685 - accuracy: 0.4575 - f1_score: 0.4588 - precision_8: 0.5708 - recall_8: 0.2098 - val_loss: 1.0706 - val_accuracy: 0.5011 - val_f1_score: 0.4877 - val_precision_8: 0.6383 - val_recall_8: 0.1782
Epoch 9/25
379/379 [==============================] - 120s 316ms/step - loss: 1.1242 - accuracy: 0.4675 - f1_score: 0.4689 - precision_8: 0.5649 - recall_8: 0.2311 - val_loss: 1.4948 - val_accuracy: 0.3096 - val_f1_score: 0.2443 - val_precision_8: 0.5864 - val_recall_8: 0.0705
Epoch 10/25
379/379 [==============================] - 121s 317ms/step - loss: 1.1157 - accuracy: 0.4765 - f1_score: 0.4774 - precision_8: 0.5864 - recall_8: 0.2405 - val_loss: 1.2469 - val_accuracy: 0.4298 - val_f1_score: 0.4210 - val_precision_8: 0.7166 - val_recall_8: 0.1990
Epoch 11/25
379/379 [==============================] - 120s 316ms/step - loss: 1.1170 - accuracy: 0.4850 - f1_score: 0.4882 - precision_8: 0.5995 - recall_8: 0.2486 - val_loss: 4.6173 - val_accuracy: 0.3326 - val_f1_score: 0.2822 - val_precision_8: 0.3353 - val_recall_8: 0.2487
Epoch 12/25
379/379 [==============================] - 120s 316ms/step - loss: 1.1000 - accuracy: 0.5009 - f1_score: 0.5038 - precision_8: 0.6317 - recall_8: 0.2832 - val_loss: 1.4532 - val_accuracy: 0.3044 - val_f1_score: 0.2175 - val_precision_8: 0.6182 - val_recall_8: 0.0757
Epoch 13/25
379/379 [==============================] - 120s 316ms/step - loss: 1.0763 - accuracy: 0.5082 - f1_score: 0.5062 - precision_8: 0.6460 - recall_8: 0.3081 - val_loss: 1.4926 - val_accuracy: 0.3645 - val_f1_score: 0.3734 - val_precision_8: 0.6068 - val_recall_8: 0.1455

# Residual with lower learning rate
checkpoint_filename = "/residual_lr0.0001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.0001,
    "optimizer": optimizers.Adam,

    "architecture": applications.ResNet50,

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline_complex(train_dataset, val_dataset, hyperparameters)

Model: "model_13"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_28 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_12 (Sequential)  (None, 180, 180, 3)       0         
                                                                 
 resnet50 (Functional)       (None, 2048)              23587712  
                                                                 
 flatten_13 (Flatten)        (None, 2048)              0         
                                                                 
 dense_26 (Dense)            (None, 256)               524544    
                                                                 
 dropout_12 (Dropout)        (None, 256)               0         
                                                                 
 dense_27 (Dense)            (None, 5)                 1285      
                                                                 
=================================================================
Total params: 24113541 (91.99 MB)
Trainable params: 24060421 (91.78 MB)
Non-trainable params: 53120 (207.50 KB)
_________________________________________________________________
Epoch 1/25
379/379 [==============================] - 92s 187ms/step - loss: 1.8010 - accuracy: 0.2936 - f1_score: 0.2922 - precision_11: 0.3472 - recall_11: 0.0440 - val_loss: 2.2952 - val_accuracy: 0.2019 - val_f1_score: 0.0672 - val_precision_11: 0.2067 - val_recall_11: 0.1982
Epoch 2/25
379/379 [==============================] - 94s 245ms/step - loss: 1.5077 - accuracy: 0.3435 - f1_score: 0.3374 - precision_11: 0.5452 - recall_11: 0.0482 - val_loss: 1.3873 - val_accuracy: 0.4031 - val_f1_score: 0.3717 - val_precision_11: 0.6380 - val_recall_11: 0.1047
Epoch 3/25
379/379 [==============================] - 119s 313ms/step - loss: 1.4634 - accuracy: 0.3728 - f1_score: 0.3644 - precision_11: 0.5318 - recall_11: 0.0744 - val_loss: 1.4730 - val_accuracy: 0.3489 - val_f1_score: 0.3265 - val_precision_11: 0.5745 - val_recall_11: 0.0401
Epoch 4/25
379/379 [==============================] - 120s 314ms/step - loss: 1.4165 - accuracy: 0.3889 - f1_score: 0.3823 - precision_11: 0.5466 - recall_11: 0.1005 - val_loss: 1.2846 - val_accuracy: 0.4573 - val_f1_score: 0.4484 - val_precision_11: 0.6944 - val_recall_11: 0.1299
Epoch 5/25
379/379 [==============================] - 119s 314ms/step - loss: 1.3653 - accuracy: 0.4147 - f1_score: 0.3975 - precision_11: 0.5879 - recall_11: 0.1395 - val_loss: 1.2614 - val_accuracy: 0.4610 - val_f1_score: 0.4034 - val_precision_11: 0.6154 - val_recall_11: 0.1841
Epoch 6/25
379/379 [==============================] - 119s 313ms/step - loss: 1.3061 - accuracy: 0.4365 - f1_score: 0.4196 - precision_11: 0.5990 - recall_11: 0.1704 - val_loss: 1.2345 - val_accuracy: 0.4425 - val_f1_score: 0.3951 - val_precision_11: 0.5884 - val_recall_11: 0.2101
Epoch 7/25
379/379 [==============================] - 119s 313ms/step - loss: 1.2435 - accuracy: 0.4605 - f1_score: 0.4391 - precision_11: 0.6171 - recall_11: 0.2146 - val_loss: 1.4265 - val_accuracy: 0.3883 - val_f1_score: 0.3574 - val_precision_11: 0.5498 - val_recall_11: 0.1722
Epoch 8/25
379/379 [==============================] - 119s 313ms/step - loss: 1.1992 - accuracy: 0.4881 - f1_score: 0.4724 - precision_11: 0.6311 - recall_11: 0.2455 - val_loss: 1.6064 - val_accuracy: 0.3846 - val_f1_score: 0.3492 - val_precision_11: 0.4481 - val_recall_11: 0.1633
Epoch 9/25
379/379 [==============================] - 119s 313ms/step - loss: 1.1374 - accuracy: 0.4986 - f1_score: 0.4946 - precision_11: 0.6222 - recall_11: 0.2718 - val_loss: 1.8489 - val_accuracy: 0.3593 - val_f1_score: 0.3065 - val_precision_11: 0.4013 - val_recall_11: 0.2806
Epoch 10/25
379/379 [==============================] - 119s 313ms/step - loss: 1.0539 - accuracy: 0.5574 - f1_score: 0.5586 - precision_11: 0.6700 - recall_11: 0.3572 - val_loss: 0.9609 - val_accuracy: 0.6050 - val_f1_score: 0.6015 - val_precision_11: 0.6609 - val_recall_11: 0.5137
Epoch 11/25
379/379 [==============================] - 119s 313ms/step - loss: 0.9706 - accuracy: 0.6086 - f1_score: 0.6098 - precision_11: 0.7319 - recall_11: 0.4464 - val_loss: 1.8337 - val_accuracy: 0.5457 - val_f1_score: 0.5355 - val_precision_11: 0.5759 - val_recall_11: 0.4900
Epoch 12/25
379/379 [==============================] - 119s 313ms/step - loss: 0.8771 - accuracy: 0.6502 - f1_score: 0.6509 - precision_11: 0.7711 - recall_11: 0.5028 - val_loss: 0.9615 - val_accuracy: 0.6444 - val_f1_score: 0.6314 - val_precision_11: 0.7375 - val_recall_11: 0.5590
Epoch 13/25
379/379 [==============================] - 119s 313ms/step - loss: 0.8046 - accuracy: 0.6823 - f1_score: 0.6825 - precision_11: 0.7920 - recall_11: 0.5565 - val_loss: 0.8075 - val_accuracy: 0.7142 - val_f1_score: 0.7078 - val_precision_11: 0.7767 - val_recall_11: 0.6429
Epoch 14/25
379/379 [==============================] - 119s 313ms/step - loss: 0.7490 - accuracy: 0.7041 - f1_score: 0.7044 - precision_11: 0.8029 - recall_11: 0.5867 - val_loss: 0.7434 - val_accuracy: 0.7238 - val_f1_score: 0.7211 - val_precision_11: 0.7823 - val_recall_11: 0.6644
Epoch 15/25
379/379 [==============================] - 83s 216ms/step - loss: 0.7064 - accuracy: 0.7270 - f1_score: 0.7273 - precision_11: 0.8196 - recall_11: 0.6194 - val_loss: 0.6393 - val_accuracy: 0.7424 - val_f1_score: 0.7429 - val_precision_11: 0.8091 - val_recall_11: 0.6763
Epoch 16/25
379/379 [==============================] - 71s 186ms/step - loss: 0.6771 - accuracy: 0.7358 - f1_score: 0.7351 - precision_11: 0.8233 - recall_11: 0.6318 - val_loss: 0.8278 - val_accuracy: 0.6793 - val_f1_score: 0.6612 - val_precision_11: 0.7402 - val_recall_11: 0.6028
Epoch 17/25
379/379 [==============================] - 71s 186ms/step - loss: 0.6212 - accuracy: 0.7628 - f1_score: 0.7621 - precision_11: 0.8379 - recall_11: 0.6754 - val_loss: 1.4363 - val_accuracy: 0.5345 - val_f1_score: 0.4814 - val_precision_11: 0.5933 - val_recall_11: 0.4722
Epoch 18/25
379/379 [==============================] - 71s 187ms/step - loss: 0.5892 - accuracy: 0.7719 - f1_score: 0.7711 - precision_11: 0.8494 - recall_11: 0.6906 - val_loss: 0.6526 - val_accuracy: 0.7751 - val_f1_score: 0.7695 - val_precision_11: 0.8289 - val_recall_11: 0.7231
Epoch 19/25
379/379 [==============================] - 71s 187ms/step - loss: 0.5745 - accuracy: 0.7801 - f1_score: 0.7793 - precision_11: 0.8514 - recall_11: 0.7031 - val_loss: 0.5654 - val_accuracy: 0.7892 - val_f1_score: 0.7839 - val_precision_11: 0.8357 - val_recall_11: 0.7327
Epoch 20/25
379/379 [==============================] - 71s 186ms/step - loss: 0.5593 - accuracy: 0.7870 - f1_score: 0.7867 - precision_11: 0.8600 - recall_11: 0.7069 - val_loss: 0.6039 - val_accuracy: 0.7788 - val_f1_score: 0.7766 - val_precision_11: 0.8123 - val_recall_11: 0.7327
Epoch 21/25
379/379 [==============================] - 71s 186ms/step - loss: 0.5310 - accuracy: 0.8009 - f1_score: 0.8005 - precision_11: 0.8674 - recall_11: 0.7271 - val_loss: 0.7652 - val_accuracy: 0.6964 - val_f1_score: 0.6705 - val_precision_11: 0.7500 - val_recall_11: 0.6526
Epoch 22/25
379/379 [==============================] - 71s 186ms/step - loss: 0.5032 - accuracy: 0.8052 - f1_score: 0.8050 - precision_11: 0.8683 - recall_11: 0.7393 - val_loss: 0.7504 - val_accuracy: 0.7223 - val_f1_score: 0.7222 - val_precision_11: 0.7429 - val_recall_11: 0.6971
Epoch 23/25
379/379 [==============================] - 71s 186ms/step - loss: 0.4851 - accuracy: 0.8163 - f1_score: 0.8163 - precision_11: 0.8785 - recall_11: 0.7529 - val_loss: 0.7957 - val_accuracy: 0.6971 - val_f1_score: 0.6884 - val_precision_11: 0.7433 - val_recall_11: 0.6578
Epoch 24/25
379/379 [==============================] - 71s 186ms/step - loss: 0.4810 - accuracy: 0.8176 - f1_score: 0.8173 - precision_11: 0.8773 - recall_11: 0.7529 - val_loss: 0.4476 - val_accuracy: 0.8270 - val_f1_score: 0.8244 - val_precision_11: 0.8629 - val_recall_11: 0.7988
Epoch 25/25
379/379 [==============================] - 71s 186ms/step - loss: 0.4497 - accuracy: 0.8294 - f1_score: 0.8293 - precision_11: 0.8816 - recall_11: 0.7734 - val_loss: 0.4660 - val_accuracy: 0.8278 - val_f1_score: 0.8252 - val_precision_11: 0.8739 - val_recall_11: 0.7765

The training using VGG16 and VGG19 was not succesful. We can attach this problem to the huge complexity of these architectures with respect to our small dataset. They are composed by a large number of parameters and they were originally trained on the ImageNet dataset, which contains millions of images across thousands of categories. A small dataset does not provide enough diverse examples for the model to learn robust features and generalize well to unseen data.

For the rest of the models in general the results were quite similar to the ones obtained when training the model from scratch (with a validation accuracy near ~0.8). However, Xception model achieved to reach the 0.9 of validation accuracy.

Note that for Inception and Residual networks we had to repeat the training because of a too high learning rate, similar as a problem occured when training the architecture from scratch.

To end with this section let's now test creating a wide model using Depthwise separable convolutions. As always, we first train the baseline model and then we apply regularization techniques.

def create_scratch_wide_model(use_batch_normalization=False, use_dropout=False, regularizer=False, data_augmentation=None):  
  inputs = keras.Input(shape=(180, 180, 3))

  if data_augmentation is not None:
    x = data_augmentation(inputs)
    x = layers.Conv2D(filters=32, kernel_size=5, use_bias=False)(x)
  else:
    x = layers.Conv2D(filters=32, kernel_size=5, use_bias=False)(inputs)

  for size in [32, 64, 128, 256, 512]:
    residual = x

    if use_batch_normalization:
      x = layers.BatchNormalization()(x)
    x = layers.Activation("relu")(x)
    x = layers.SeparableConv2D(size, 3, padding="same", use_bias=False)(x)
  
    if use_batch_normalization:
      x = layers.BatchNormalization()(x)
    x = layers.Activation("relu")(x)
    x = layers.SeparableConv2D(size, 3, padding="same", use_bias=False)(x)
  
    x = layers.MaxPooling2D(3, strides=2, padding="same")(x)
  
    residual = layers.Conv2D(size, 1, strides=2, padding="same", use_bias=False)(residual)
    x = layers.add([x, residual])

  x = layers.GlobalAveragePooling2D()(x)

  if (use_dropout):
    x = layers.Dropout(0.5)(x)
  outputs = layers.Dense(5, activation="softmax")(x)

  model = keras.Model(inputs, outputs)

  model.summary()
  
  return model

def train_model(model, train_dataset, val_dataset, epochs=20, lr=0.001, optimizer=keras.optimizers.Adam, checkpoint=None, lr_scheduler=None, early_stop=None):

  model.compile(optimizer=optimizer(learning_rate=lr),
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
  )

  callbacks = []
  if checkpoint is not None:
    callbacks.append(checkpoint)
  if lr_scheduler is not None:
    callbacks.append(lr_scheduler)
  if early_stop is not None:
    callbacks.append(early_stop)

  history = model.fit(train_dataset, epochs=epochs, validation_data=val_dataset, callbacks=callbacks)
  return history

def execute_pipeline(train_dataset, val_dataset, hyperparameters):

  # Set the seed to ensure reproducibility
  keras.utils.set_random_seed(RANDOM_SEED)

  model = create_scratch_wide_model(use_batch_normalization=hyperparameters["bn"], use_dropout=hyperparameters["dropout"], data_augmentation=hyperparameters["augmentation"], regularizer=hyperparameters["regularizer"])
  history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
  plot(history)

  return model, history

# Scratch wide model
checkpoint_filename = "/cnn_wide_lr0.001.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": None,
    "dropout": False,
    "augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_1 (InputLayer)        [(None, 180, 180, 3)]        0         []                            
                                                                                                  
 conv2d (Conv2D)             (None, 176, 176, 32)         2400      ['input_1[0][0]']             
                                                                                                  
 activation (Activation)     (None, 176, 176, 32)         0         ['conv2d[0][0]']              
                                                                                                  
 separable_conv2d (Separabl  (None, 176, 176, 32)         1312      ['activation[0][0]']          
 eConv2D)                                                                                         
                                                                                                  
 activation_1 (Activation)   (None, 176, 176, 32)         0         ['separable_conv2d[0][0]']    
                                                                                                  
 separable_conv2d_1 (Separa  (None, 176, 176, 32)         1312      ['activation_1[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d (MaxPooling2  (None, 88, 88, 32)           0         ['separable_conv2d_1[0][0]']  
 D)                                                                                               
                                                                                                  
 conv2d_1 (Conv2D)           (None, 88, 88, 32)           1024      ['conv2d[0][0]']              
                                                                                                  
 add (Add)                   (None, 88, 88, 32)           0         ['max_pooling2d[0][0]',       
                                                                     'conv2d_1[0][0]']            
                                                                                                  
 activation_2 (Activation)   (None, 88, 88, 32)           0         ['add[0][0]']                 
                                                                                                  
 separable_conv2d_2 (Separa  (None, 88, 88, 64)           2336      ['activation_2[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 activation_3 (Activation)   (None, 88, 88, 64)           0         ['separable_conv2d_2[0][0]']  
                                                                                                  
 separable_conv2d_3 (Separa  (None, 88, 88, 64)           4672      ['activation_3[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_1 (MaxPoolin  (None, 44, 44, 64)           0         ['separable_conv2d_3[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_2 (Conv2D)           (None, 44, 44, 64)           2048      ['add[0][0]']                 
                                                                                                  
 add_1 (Add)                 (None, 44, 44, 64)           0         ['max_pooling2d_1[0][0]',     
                                                                     'conv2d_2[0][0]']            
                                                                                                  
 activation_4 (Activation)   (None, 44, 44, 64)           0         ['add_1[0][0]']               
                                                                                                  
 separable_conv2d_4 (Separa  (None, 44, 44, 128)          8768      ['activation_4[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 activation_5 (Activation)   (None, 44, 44, 128)          0         ['separable_conv2d_4[0][0]']  
                                                                                                  
 separable_conv2d_5 (Separa  (None, 44, 44, 128)          17536     ['activation_5[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_2 (MaxPoolin  (None, 22, 22, 128)          0         ['separable_conv2d_5[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_3 (Conv2D)           (None, 22, 22, 128)          8192      ['add_1[0][0]']               
                                                                                                  
 add_2 (Add)                 (None, 22, 22, 128)          0         ['max_pooling2d_2[0][0]',     
                                                                     'conv2d_3[0][0]']            
                                                                                                  
 activation_6 (Activation)   (None, 22, 22, 128)          0         ['add_2[0][0]']               
                                                                                                  
 separable_conv2d_6 (Separa  (None, 22, 22, 256)          33920     ['activation_6[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 activation_7 (Activation)   (None, 22, 22, 256)          0         ['separable_conv2d_6[0][0]']  
                                                                                                  
 separable_conv2d_7 (Separa  (None, 22, 22, 256)          67840     ['activation_7[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_3 (MaxPoolin  (None, 11, 11, 256)          0         ['separable_conv2d_7[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_4 (Conv2D)           (None, 11, 11, 256)          32768     ['add_2[0][0]']               
                                                                                                  
 add_3 (Add)                 (None, 11, 11, 256)          0         ['max_pooling2d_3[0][0]',     
                                                                     'conv2d_4[0][0]']            
                                                                                                  
 activation_8 (Activation)   (None, 11, 11, 256)          0         ['add_3[0][0]']               
                                                                                                  
 separable_conv2d_8 (Separa  (None, 11, 11, 512)          133376    ['activation_8[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 activation_9 (Activation)   (None, 11, 11, 512)          0         ['separable_conv2d_8[0][0]']  
                                                                                                  
 separable_conv2d_9 (Separa  (None, 11, 11, 512)          266752    ['activation_9[0][0]']        
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_4 (MaxPoolin  (None, 6, 6, 512)            0         ['separable_conv2d_9[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_5 (Conv2D)           (None, 6, 6, 512)            131072    ['add_3[0][0]']               
                                                                                                  
 add_4 (Add)                 (None, 6, 6, 512)            0         ['max_pooling2d_4[0][0]',     
                                                                     'conv2d_5[0][0]']            
                                                                                                  
 global_average_pooling2d (  (None, 512)                  0         ['add_4[0][0]']               
 GlobalAveragePooling2D)                                                                          
                                                                                                  
 dense (Dense)               (None, 5)                    2565      ['global_average_pooling2d[0][
                                                                    0]']                          
                                                                                                  
==================================================================================================
Total params: 717893 (2.74 MB)
Trainable params: 717893 (2.74 MB)
Non-trainable params: 0 (0.00 Byte)
__________________________________________________________________________________________________
Epoch 1/25

2024-02-28 08:54:28.709679: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-28 08:54:33.766282: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f9729a93310 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-28 08:54:33.766315: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-28 08:54:33.778649: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1709106873.883484    3259 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 55s 97ms/step - loss: 1.5470 - accuracy: 0.3008 - f1_score: 0.2913 - precision: 0.4838 - recall: 0.0172 - val_loss: 1.5439 - val_accuracy: 0.3170 - val_f1_score: 0.2820 - val_precision: 0.4516 - val_recall: 0.0208
Epoch 2/25
379/379 [==============================] - 29s 74ms/step - loss: 1.3126 - accuracy: 0.4446 - f1_score: 0.4422 - precision: 0.6046 - recall: 0.1682 - val_loss: 1.0308 - val_accuracy: 0.5657 - val_f1_score: 0.5525 - val_precision: 0.6348 - val_recall: 0.4439
Epoch 3/25
379/379 [==============================] - 28s 74ms/step - loss: 0.9037 - accuracy: 0.6292 - f1_score: 0.6275 - precision: 0.7150 - recall: 0.4762 - val_loss: 0.7669 - val_accuracy: 0.7053 - val_f1_score: 0.7073 - val_precision: 0.7749 - val_recall: 0.5828
Epoch 4/25
379/379 [==============================] - 34s 88ms/step - loss: 0.7309 - accuracy: 0.7031 - f1_score: 0.7017 - precision: 0.7766 - recall: 0.6068 - val_loss: 0.6557 - val_accuracy: 0.7365 - val_f1_score: 0.7324 - val_precision: 0.7846 - val_recall: 0.6600
Epoch 5/25
379/379 [==============================] - 53s 138ms/step - loss: 0.6239 - accuracy: 0.7507 - f1_score: 0.7499 - precision: 0.8031 - recall: 0.6815 - val_loss: 0.6647 - val_accuracy: 0.7327 - val_f1_score: 0.7399 - val_precision: 0.7811 - val_recall: 0.6704
Epoch 6/25
379/379 [==============================] - 51s 134ms/step - loss: 0.5460 - accuracy: 0.7859 - f1_score: 0.7855 - precision: 0.8307 - recall: 0.7313 - val_loss: 0.5111 - val_accuracy: 0.8055 - val_f1_score: 0.8060 - val_precision: 0.8301 - val_recall: 0.7728
Epoch 7/25
379/379 [==============================] - 52s 135ms/step - loss: 0.4690 - accuracy: 0.8164 - f1_score: 0.8162 - precision: 0.8523 - recall: 0.7781 - val_loss: 0.5556 - val_accuracy: 0.7758 - val_f1_score: 0.7758 - val_precision: 0.8056 - val_recall: 0.7231
Epoch 8/25
379/379 [==============================] - 52s 135ms/step - loss: 0.4256 - accuracy: 0.8371 - f1_score: 0.8369 - precision: 0.8674 - recall: 0.8032 - val_loss: 0.4527 - val_accuracy: 0.8278 - val_f1_score: 0.8280 - val_precision: 0.8566 - val_recall: 0.8025
Epoch 9/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3780 - accuracy: 0.8525 - f1_score: 0.8524 - precision: 0.8801 - recall: 0.8294 - val_loss: 0.5070 - val_accuracy: 0.8189 - val_f1_score: 0.8183 - val_precision: 0.8356 - val_recall: 0.7921
Epoch 10/25
379/379 [==============================] - 52s 135ms/step - loss: 0.3499 - accuracy: 0.8639 - f1_score: 0.8639 - precision: 0.8864 - recall: 0.8387 - val_loss: 0.4360 - val_accuracy: 0.8337 - val_f1_score: 0.8337 - val_precision: 0.8562 - val_recall: 0.8048
Epoch 11/25
379/379 [==============================] - 43s 111ms/step - loss: 0.3101 - accuracy: 0.8786 - f1_score: 0.8786 - precision: 0.8984 - recall: 0.8573 - val_loss: 0.4203 - val_accuracy: 0.8478 - val_f1_score: 0.8486 - val_precision: 0.8685 - val_recall: 0.8241
Epoch 12/25
379/379 [==============================] - 51s 134ms/step - loss: 0.2733 - accuracy: 0.8967 - f1_score: 0.8967 - precision: 0.9145 - recall: 0.8789 - val_loss: 0.4267 - val_accuracy: 0.8552 - val_f1_score: 0.8561 - val_precision: 0.8648 - val_recall: 0.8456
Epoch 13/25
379/379 [==============================] - 51s 134ms/step - loss: 0.2617 - accuracy: 0.9043 - f1_score: 0.9043 - precision: 0.9167 - recall: 0.8890 - val_loss: 0.4600 - val_accuracy: 0.8367 - val_f1_score: 0.8353 - val_precision: 0.8519 - val_recall: 0.8241
Epoch 14/25
379/379 [==============================] - 51s 134ms/step - loss: 0.2284 - accuracy: 0.9114 - f1_score: 0.9115 - precision: 0.9235 - recall: 0.9010 - val_loss: 0.4604 - val_accuracy: 0.8530 - val_f1_score: 0.8525 - val_precision: 0.8678 - val_recall: 0.8434
Epoch 15/25
379/379 [==============================] - 52s 135ms/step - loss: 0.2180 - accuracy: 0.9182 - f1_score: 0.9182 - precision: 0.9286 - recall: 0.9089 - val_loss: 0.5703 - val_accuracy: 0.8389 - val_f1_score: 0.8395 - val_precision: 0.8460 - val_recall: 0.8322
Epoch 16/25
379/379 [==============================] - 52s 135ms/step - loss: 0.1798 - accuracy: 0.9316 - f1_score: 0.9316 - precision: 0.9399 - recall: 0.9254 - val_loss: 0.4965 - val_accuracy: 0.8493 - val_f1_score: 0.8488 - val_precision: 0.8556 - val_recall: 0.8404

# Scratch wide model with data augmentation
checkpoint_filename = "/cnn_wide_lr0.001_da.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": None,
    "dropout": False,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_1"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_2 (InputLayer)        [(None, 180, 180, 3)]        0         []                            
                                                                                                  
 augmentation (Sequential)   (None, 180, 180, 3)          0         ['input_2[0][0]']             
                                                                                                  
 conv2d_6 (Conv2D)           (None, 176, 176, 32)         2400      ['augmentation[0][0]']        
                                                                                                  
 activation_10 (Activation)  (None, 176, 176, 32)         0         ['conv2d_6[0][0]']            
                                                                                                  
 separable_conv2d_10 (Separ  (None, 176, 176, 32)         1312      ['activation_10[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 activation_11 (Activation)  (None, 176, 176, 32)         0         ['separable_conv2d_10[0][0]'] 
                                                                                                  
 separable_conv2d_11 (Separ  (None, 176, 176, 32)         1312      ['activation_11[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 max_pooling2d_5 (MaxPoolin  (None, 88, 88, 32)           0         ['separable_conv2d_11[0][0]'] 
 g2D)                                                                                             
                                                                                                  
 conv2d_7 (Conv2D)           (None, 88, 88, 32)           1024      ['conv2d_6[0][0]']            
                                                                                                  
 add_5 (Add)                 (None, 88, 88, 32)           0         ['max_pooling2d_5[0][0]',     
                                                                     'conv2d_7[0][0]']            
                                                                                                  
 activation_12 (Activation)  (None, 88, 88, 32)           0         ['add_5[0][0]']               
                                                                                                  
 separable_conv2d_12 (Separ  (None, 88, 88, 64)           2336      ['activation_12[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 activation_13 (Activation)  (None, 88, 88, 64)           0         ['separable_conv2d_12[0][0]'] 
                                                                                                  
 separable_conv2d_13 (Separ  (None, 88, 88, 64)           4672      ['activation_13[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 max_pooling2d_6 (MaxPoolin  (None, 44, 44, 64)           0         ['separable_conv2d_13[0][0]'] 
 g2D)                                                                                             
                                                                                                  
 conv2d_8 (Conv2D)           (None, 44, 44, 64)           2048      ['add_5[0][0]']               
                                                                                                  
 add_6 (Add)                 (None, 44, 44, 64)           0         ['max_pooling2d_6[0][0]',     
                                                                     'conv2d_8[0][0]']            
                                                                                                  
 activation_14 (Activation)  (None, 44, 44, 64)           0         ['add_6[0][0]']               
                                                                                                  
 separable_conv2d_14 (Separ  (None, 44, 44, 128)          8768      ['activation_14[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 activation_15 (Activation)  (None, 44, 44, 128)          0         ['separable_conv2d_14[0][0]'] 
                                                                                                  
 separable_conv2d_15 (Separ  (None, 44, 44, 128)          17536     ['activation_15[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 max_pooling2d_7 (MaxPoolin  (None, 22, 22, 128)          0         ['separable_conv2d_15[0][0]'] 
 g2D)                                                                                             
                                                                                                  
 conv2d_9 (Conv2D)           (None, 22, 22, 128)          8192      ['add_6[0][0]']               
                                                                                                  
 add_7 (Add)                 (None, 22, 22, 128)          0         ['max_pooling2d_7[0][0]',     
                                                                     'conv2d_9[0][0]']            
                                                                                                  
 activation_16 (Activation)  (None, 22, 22, 128)          0         ['add_7[0][0]']               
                                                                                                  
 separable_conv2d_16 (Separ  (None, 22, 22, 256)          33920     ['activation_16[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 activation_17 (Activation)  (None, 22, 22, 256)          0         ['separable_conv2d_16[0][0]'] 
                                                                                                  
 separable_conv2d_17 (Separ  (None, 22, 22, 256)          67840     ['activation_17[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 max_pooling2d_8 (MaxPoolin  (None, 11, 11, 256)          0         ['separable_conv2d_17[0][0]'] 
 g2D)                                                                                             
                                                                                                  
 conv2d_10 (Conv2D)          (None, 11, 11, 256)          32768     ['add_7[0][0]']               
                                                                                                  
 add_8 (Add)                 (None, 11, 11, 256)          0         ['max_pooling2d_8[0][0]',     
                                                                     'conv2d_10[0][0]']           
                                                                                                  
 activation_18 (Activation)  (None, 11, 11, 256)          0         ['add_8[0][0]']               
                                                                                                  
 separable_conv2d_18 (Separ  (None, 11, 11, 512)          133376    ['activation_18[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 activation_19 (Activation)  (None, 11, 11, 512)          0         ['separable_conv2d_18[0][0]'] 
                                                                                                  
 separable_conv2d_19 (Separ  (None, 11, 11, 512)          266752    ['activation_19[0][0]']       
 ableConv2D)                                                                                      
                                                                                                  
 max_pooling2d_9 (MaxPoolin  (None, 6, 6, 512)            0         ['separable_conv2d_19[0][0]'] 
 g2D)                                                                                             
                                                                                                  
 conv2d_11 (Conv2D)          (None, 6, 6, 512)            131072    ['add_8[0][0]']               
                                                                                                  
 add_9 (Add)                 (None, 6, 6, 512)            0         ['max_pooling2d_9[0][0]',     
                                                                     'conv2d_11[0][0]']           
                                                                                                  
 global_average_pooling2d_1  (None, 512)                  0         ['add_9[0][0]']               
  (GlobalAveragePooling2D)                                                                        
                                                                                                  
 dense_1 (Dense)             (None, 5)                    2565      ['global_average_pooling2d_1[0
                                                                    ][0]']                        
                                                                                                  
==================================================================================================
Total params: 717893 (2.74 MB)
Trainable params: 717893 (2.74 MB)
Non-trainable params: 0 (0.00 Byte)
__________________________________________________________________________________________________
Epoch 1/25
379/379 [==============================] - 58s 143ms/step - loss: 1.5808 - accuracy: 0.2696 - f1_score: 0.2472 - precision_1: 0.5359 - recall_1: 0.0068 - val_loss: 1.5236 - val_accuracy: 0.3571 - val_f1_score: 0.3294 - val_precision_1: 0.5581 - val_recall_1: 0.0178
Epoch 2/25
379/379 [==============================] - 54s 142ms/step - loss: 1.4438 - accuracy: 0.3697 - f1_score: 0.3668 - precision_1: 0.5330 - recall_1: 0.0599 - val_loss: 1.3422 - val_accuracy: 0.4165 - val_f1_score: 0.3924 - val_precision_1: 0.6314 - val_recall_1: 0.1106
Epoch 3/25
379/379 [==============================] - 55s 144ms/step - loss: 1.2150 - accuracy: 0.5082 - f1_score: 0.5081 - precision_1: 0.6424 - recall_1: 0.2682 - val_loss: 0.9925 - val_accuracy: 0.6110 - val_f1_score: 0.6041 - val_precision_1: 0.6857 - val_recall_1: 0.4714
Epoch 4/25
379/379 [==============================] - 55s 144ms/step - loss: 0.9027 - accuracy: 0.6559 - f1_score: 0.6551 - precision_1: 0.7468 - recall_1: 0.5208 - val_loss: 0.6972 - val_accuracy: 0.7320 - val_f1_score: 0.7289 - val_precision_1: 0.7799 - val_recall_1: 0.6578
Epoch 5/25
379/379 [==============================] - 55s 144ms/step - loss: 0.6998 - accuracy: 0.7374 - f1_score: 0.7371 - precision_1: 0.8029 - recall_1: 0.6562 - val_loss: 0.5645 - val_accuracy: 0.7958 - val_f1_score: 0.7964 - val_precision_1: 0.8519 - val_recall_1: 0.7342
Epoch 6/25
379/379 [==============================] - 55s 144ms/step - loss: 0.5854 - accuracy: 0.7803 - f1_score: 0.7801 - precision_1: 0.8311 - recall_1: 0.7200 - val_loss: 0.4997 - val_accuracy: 0.8129 - val_f1_score: 0.8163 - val_precision_1: 0.8573 - val_recall_1: 0.7713
Epoch 7/25
379/379 [==============================] - 55s 144ms/step - loss: 0.5121 - accuracy: 0.8094 - f1_score: 0.8093 - precision_1: 0.8514 - recall_1: 0.7590 - val_loss: 0.4708 - val_accuracy: 0.8263 - val_f1_score: 0.8264 - val_precision_1: 0.8567 - val_recall_1: 0.7765
Epoch 8/25
379/379 [==============================] - 45s 116ms/step - loss: 0.4818 - accuracy: 0.8216 - f1_score: 0.8215 - precision_1: 0.8612 - recall_1: 0.7770 - val_loss: 0.4180 - val_accuracy: 0.8448 - val_f1_score: 0.8431 - val_precision_1: 0.8804 - val_recall_1: 0.8196
Epoch 9/25
379/379 [==============================] - 43s 113ms/step - loss: 0.4371 - accuracy: 0.8357 - f1_score: 0.8358 - precision_1: 0.8693 - recall_1: 0.7966 - val_loss: 0.3905 - val_accuracy: 0.8575 - val_f1_score: 0.8575 - val_precision_1: 0.8859 - val_recall_1: 0.8300
Epoch 10/25
379/379 [==============================] - 55s 143ms/step - loss: 0.4059 - accuracy: 0.8526 - f1_score: 0.8527 - precision_1: 0.8837 - recall_1: 0.8165 - val_loss: 0.3706 - val_accuracy: 0.8634 - val_f1_score: 0.8625 - val_precision_1: 0.8862 - val_recall_1: 0.8382
Epoch 11/25
379/379 [==============================] - 54s 142ms/step - loss: 0.3799 - accuracy: 0.8588 - f1_score: 0.8589 - precision_1: 0.8883 - recall_1: 0.8277 - val_loss: 0.3351 - val_accuracy: 0.8723 - val_f1_score: 0.8717 - val_precision_1: 0.8901 - val_recall_1: 0.8537
Epoch 12/25
379/379 [==============================] - 54s 142ms/step - loss: 0.3636 - accuracy: 0.8632 - f1_score: 0.8633 - precision_1: 0.8908 - recall_1: 0.8355 - val_loss: 0.3214 - val_accuracy: 0.8731 - val_f1_score: 0.8742 - val_precision_1: 0.8923 - val_recall_1: 0.8486
Epoch 13/25
379/379 [==============================] - 55s 143ms/step - loss: 0.3385 - accuracy: 0.8728 - f1_score: 0.8729 - precision_1: 0.8968 - recall_1: 0.8468 - val_loss: 0.3537 - val_accuracy: 0.8723 - val_f1_score: 0.8720 - val_precision_1: 0.8840 - val_recall_1: 0.8426
Epoch 14/25
379/379 [==============================] - 55s 143ms/step - loss: 0.3159 - accuracy: 0.8841 - f1_score: 0.8843 - precision_1: 0.9042 - recall_1: 0.8602 - val_loss: 0.3680 - val_accuracy: 0.8649 - val_f1_score: 0.8626 - val_precision_1: 0.8867 - val_recall_1: 0.8486
Epoch 15/25
379/379 [==============================] - 55s 143ms/step - loss: 0.3073 - accuracy: 0.8848 - f1_score: 0.8850 - precision_1: 0.9048 - recall_1: 0.8606 - val_loss: 0.3152 - val_accuracy: 0.8872 - val_f1_score: 0.8879 - val_precision_1: 0.9018 - val_recall_1: 0.8723
Epoch 16/25
379/379 [==============================] - 55s 143ms/step - loss: 0.2992 - accuracy: 0.8855 - f1_score: 0.8856 - precision_1: 0.9057 - recall_1: 0.8683 - val_loss: 0.3537 - val_accuracy: 0.8671 - val_f1_score: 0.8682 - val_precision_1: 0.8838 - val_recall_1: 0.8411
Epoch 17/25
379/379 [==============================] - 55s 143ms/step - loss: 0.2777 - accuracy: 0.8984 - f1_score: 0.8985 - precision_1: 0.9147 - recall_1: 0.8789 - val_loss: 0.2691 - val_accuracy: 0.9027 - val_f1_score: 0.9038 - val_precision_1: 0.9152 - val_recall_1: 0.8894
Epoch 18/25
379/379 [==============================] - 55s 143ms/step - loss: 0.2576 - accuracy: 0.9020 - f1_score: 0.9022 - precision_1: 0.9172 - recall_1: 0.8869 - val_loss: 0.2693 - val_accuracy: 0.9013 - val_f1_score: 0.9022 - val_precision_1: 0.9190 - val_recall_1: 0.8924
Epoch 19/25
379/379 [==============================] - 55s 144ms/step - loss: 0.2721 - accuracy: 0.9006 - f1_score: 0.9007 - precision_1: 0.9151 - recall_1: 0.8850 - val_loss: 0.2721 - val_accuracy: 0.8998 - val_f1_score: 0.9005 - val_precision_1: 0.9153 - val_recall_1: 0.8820
Epoch 20/25
379/379 [==============================] - 55s 143ms/step - loss: 0.2454 - accuracy: 0.9104 - f1_score: 0.9106 - precision_1: 0.9252 - recall_1: 0.8930 - val_loss: 0.2727 - val_accuracy: 0.8983 - val_f1_score: 0.8998 - val_precision_1: 0.9156 - val_recall_1: 0.8782
Epoch 21/25
379/379 [==============================] - 55s 144ms/step - loss: 0.2459 - accuracy: 0.9073 - f1_score: 0.9074 - precision_1: 0.9228 - recall_1: 0.8925 - val_loss: 0.2803 - val_accuracy: 0.9094 - val_f1_score: 0.9097 - val_precision_1: 0.9190 - val_recall_1: 0.8931
Epoch 22/25
379/379 [==============================] - 34s 88ms/step - loss: 0.2345 - accuracy: 0.9114 - f1_score: 0.9115 - precision_1: 0.9240 - recall_1: 0.8987 - val_loss: 0.3119 - val_accuracy: 0.8946 - val_f1_score: 0.8949 - val_precision_1: 0.9118 - val_recall_1: 0.8745

# Scratch wide model with data augmentation and dropout
checkpoint_filename = "/cnn_wide_lr0.001_dropout_da.keras"
hyperparameters = {
    "epochs": 25,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "bn": False,
    "regularizer": None,
    "dropout": True,
    "augmentation": keras.Sequential(
      [
        layers.RandomFlip("horizontal"),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.2)
      ], name="augmentation"),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(train_dataset, val_dataset, hyperparameters)

Model: "model_2"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_4 (InputLayer)        [(None, 180, 180, 3)]        0         []                            
                                                                                                  
 augmentation (Sequential)   (None, 180, 180, 3)          0         ['input_4[0][0]']             
                                                                                                  
 conv2d_211 (Conv2D)         (None, 176, 176, 32)         2400      ['augmentation[0][0]']        
                                                                                                  
 activation_203 (Activation  (None, 176, 176, 32)         0         ['conv2d_211[0][0]']          
 )                                                                                                
                                                                                                  
 separable_conv2d (Separabl  (None, 176, 176, 32)         1312      ['activation_203[0][0]']      
 eConv2D)                                                                                         
                                                                                                  
 activation_204 (Activation  (None, 176, 176, 32)         0         ['separable_conv2d[0][0]']    
 )                                                                                                
                                                                                                  
 separable_conv2d_1 (Separa  (None, 176, 176, 32)         1312      ['activation_204[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_7 (MaxPoolin  (None, 88, 88, 32)           0         ['separable_conv2d_1[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_212 (Conv2D)         (None, 88, 88, 32)           1024      ['conv2d_211[0][0]']          
                                                                                                  
 add (Add)                   (None, 88, 88, 32)           0         ['max_pooling2d_7[0][0]',     
                                                                     'conv2d_212[0][0]']          
                                                                                                  
 activation_205 (Activation  (None, 88, 88, 32)           0         ['add[0][0]']                 
 )                                                                                                
                                                                                                  
 separable_conv2d_2 (Separa  (None, 88, 88, 64)           2336      ['activation_205[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 activation_206 (Activation  (None, 88, 88, 64)           0         ['separable_conv2d_2[0][0]']  
 )                                                                                                
                                                                                                  
 separable_conv2d_3 (Separa  (None, 88, 88, 64)           4672      ['activation_206[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_8 (MaxPoolin  (None, 44, 44, 64)           0         ['separable_conv2d_3[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_213 (Conv2D)         (None, 44, 44, 64)           2048      ['add[0][0]']                 
                                                                                                  
 add_1 (Add)                 (None, 44, 44, 64)           0         ['max_pooling2d_8[0][0]',     
                                                                     'conv2d_213[0][0]']          
                                                                                                  
 activation_207 (Activation  (None, 44, 44, 64)           0         ['add_1[0][0]']               
 )                                                                                                
                                                                                                  
 separable_conv2d_4 (Separa  (None, 44, 44, 128)          8768      ['activation_207[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 activation_208 (Activation  (None, 44, 44, 128)          0         ['separable_conv2d_4[0][0]']  
 )                                                                                                
                                                                                                  
 separable_conv2d_5 (Separa  (None, 44, 44, 128)          17536     ['activation_208[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_9 (MaxPoolin  (None, 22, 22, 128)          0         ['separable_conv2d_5[0][0]']  
 g2D)                                                                                             
                                                                                                  
 conv2d_214 (Conv2D)         (None, 22, 22, 128)          8192      ['add_1[0][0]']               
                                                                                                  
 add_2 (Add)                 (None, 22, 22, 128)          0         ['max_pooling2d_9[0][0]',     
                                                                     'conv2d_214[0][0]']          
                                                                                                  
 activation_209 (Activation  (None, 22, 22, 128)          0         ['add_2[0][0]']               
 )                                                                                                
                                                                                                  
 separable_conv2d_6 (Separa  (None, 22, 22, 256)          33920     ['activation_209[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 activation_210 (Activation  (None, 22, 22, 256)          0         ['separable_conv2d_6[0][0]']  
 )                                                                                                
                                                                                                  
 separable_conv2d_7 (Separa  (None, 22, 22, 256)          67840     ['activation_210[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_10 (MaxPooli  (None, 11, 11, 256)          0         ['separable_conv2d_7[0][0]']  
 ng2D)                                                                                            
                                                                                                  
 conv2d_215 (Conv2D)         (None, 11, 11, 256)          32768     ['add_2[0][0]']               
                                                                                                  
 add_3 (Add)                 (None, 11, 11, 256)          0         ['max_pooling2d_10[0][0]',    
                                                                     'conv2d_215[0][0]']          
                                                                                                  
 activation_211 (Activation  (None, 11, 11, 256)          0         ['add_3[0][0]']               
 )                                                                                                
                                                                                                  
 separable_conv2d_8 (Separa  (None, 11, 11, 512)          133376    ['activation_211[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 activation_212 (Activation  (None, 11, 11, 512)          0         ['separable_conv2d_8[0][0]']  
 )                                                                                                
                                                                                                  
 separable_conv2d_9 (Separa  (None, 11, 11, 512)          266752    ['activation_212[0][0]']      
 bleConv2D)                                                                                       
                                                                                                  
 max_pooling2d_11 (MaxPooli  (None, 6, 6, 512)            0         ['separable_conv2d_9[0][0]']  
 ng2D)                                                                                            
                                                                                                  
 conv2d_216 (Conv2D)         (None, 6, 6, 512)            131072    ['add_3[0][0]']               
                                                                                                  
 add_4 (Add)                 (None, 6, 6, 512)            0         ['max_pooling2d_11[0][0]',    
                                                                     'conv2d_216[0][0]']          
                                                                                                  
 global_average_pooling2d (  (None, 512)                  0         ['add_4[0][0]']               
 GlobalAveragePooling2D)                                                                          
                                                                                                  
 dropout_2 (Dropout)         (None, 512)                  0         ['global_average_pooling2d[0][
                                                                    0]']                          
                                                                                                  
 dense_4 (Dense)             (None, 5)                    2565      ['dropout_2[0][0]']           
                                                                                                  
==================================================================================================
Total params: 717893 (2.74 MB)
Trainable params: 717893 (2.74 MB)
Non-trainable params: 0 (0.00 Byte)
__________________________________________________________________________________________________
Epoch 1/25
379/379 [==============================] - 49s 102ms/step - loss: 1.5719 - accuracy: 0.2710 - f1_score: 0.2593 - precision_3: 0.5124 - recall_3: 0.0120 - val_loss: 1.4883 - val_accuracy: 0.3549 - val_f1_score: 0.3146 - val_precision_3: 0.5045 - val_recall_3: 0.0416
Epoch 2/25
379/379 [==============================] - 32s 82ms/step - loss: 1.4439 - accuracy: 0.3685 - f1_score: 0.3655 - precision_3: 0.5220 - recall_3: 0.0675 - val_loss: 1.4304 - val_accuracy: 0.3912 - val_f1_score: 0.3730 - val_precision_3: 0.5257 - val_recall_3: 0.1292
Epoch 3/25
379/379 [==============================] - 32s 82ms/step - loss: 1.2483 - accuracy: 0.4918 - f1_score: 0.4901 - precision_3: 0.6257 - recall_3: 0.2400 - val_loss: 1.1291 - val_accuracy: 0.5382 - val_f1_score: 0.5104 - val_precision_3: 0.6586 - val_recall_3: 0.3252
Epoch 4/25
379/379 [==============================] - 32s 82ms/step - loss: 1.0053 - accuracy: 0.6008 - f1_score: 0.5992 - precision_3: 0.7046 - recall_3: 0.4335 - val_loss: 0.8609 - val_accuracy: 0.6630 - val_f1_score: 0.6540 - val_precision_3: 0.7302 - val_recall_3: 0.5546
Epoch 5/25
379/379 [==============================] - 32s 82ms/step - loss: 0.8318 - accuracy: 0.6660 - f1_score: 0.6654 - precision_3: 0.7492 - recall_3: 0.5563 - val_loss: 0.6484 - val_accuracy: 0.7454 - val_f1_score: 0.7467 - val_precision_3: 0.7874 - val_recall_3: 0.6793
Epoch 6/25
379/379 [==============================] - 32s 82ms/step - loss: 0.6879 - accuracy: 0.7288 - f1_score: 0.7284 - precision_3: 0.7921 - recall_3: 0.6519 - val_loss: 0.6067 - val_accuracy: 0.7661 - val_f1_score: 0.7656 - val_precision_3: 0.7958 - val_recall_3: 0.7290
Epoch 7/25
379/379 [==============================] - 32s 82ms/step - loss: 0.6002 - accuracy: 0.7692 - f1_score: 0.7689 - precision_3: 0.8181 - recall_3: 0.7080 - val_loss: 0.5430 - val_accuracy: 0.7988 - val_f1_score: 0.7969 - val_precision_3: 0.8373 - val_recall_3: 0.7565
Epoch 8/25
379/379 [==============================] - 32s 82ms/step - loss: 0.5423 - accuracy: 0.7901 - f1_score: 0.7902 - precision_3: 0.8336 - recall_3: 0.7355 - val_loss: 0.5093 - val_accuracy: 0.8129 - val_f1_score: 0.8104 - val_precision_3: 0.8579 - val_recall_3: 0.7706
Epoch 9/25
379/379 [==============================] - 32s 82ms/step - loss: 0.4798 - accuracy: 0.8168 - f1_score: 0.8169 - precision_3: 0.8567 - recall_3: 0.7710 - val_loss: 0.4702 - val_accuracy: 0.8151 - val_f1_score: 0.8141 - val_precision_3: 0.8479 - val_recall_3: 0.7906
Epoch 10/25
379/379 [==============================] - 32s 82ms/step - loss: 0.4629 - accuracy: 0.8216 - f1_score: 0.8218 - precision_3: 0.8620 - recall_3: 0.7804 - val_loss: 0.4017 - val_accuracy: 0.8419 - val_f1_score: 0.8408 - val_precision_3: 0.8719 - val_recall_3: 0.8137
Epoch 11/25
379/379 [==============================] - 31s 82ms/step - loss: 0.4090 - accuracy: 0.8445 - f1_score: 0.8446 - precision_3: 0.8746 - recall_3: 0.8129 - val_loss: 0.4890 - val_accuracy: 0.8263 - val_f1_score: 0.8226 - val_precision_3: 0.8503 - val_recall_3: 0.8055
Epoch 12/25
379/379 [==============================] - 32s 83ms/step - loss: 0.3887 - accuracy: 0.8512 - f1_score: 0.8512 - precision_3: 0.8798 - recall_3: 0.8206 - val_loss: 0.3347 - val_accuracy: 0.8716 - val_f1_score: 0.8733 - val_precision_3: 0.9013 - val_recall_3: 0.8478
Epoch 13/25
379/379 [==============================] - 32s 82ms/step - loss: 0.3696 - accuracy: 0.8587 - f1_score: 0.8590 - precision_3: 0.8862 - recall_3: 0.8312 - val_loss: 0.3229 - val_accuracy: 0.8731 - val_f1_score: 0.8730 - val_precision_3: 0.8931 - val_recall_3: 0.8560
Epoch 14/25
379/379 [==============================] - 32s 82ms/step - loss: 0.3380 - accuracy: 0.8700 - f1_score: 0.8702 - precision_3: 0.8953 - recall_3: 0.8488 - val_loss: 0.3045 - val_accuracy: 0.8782 - val_f1_score: 0.8794 - val_precision_3: 0.9079 - val_recall_3: 0.8560
Epoch 15/25
379/379 [==============================] - 32s 82ms/step - loss: 0.3268 - accuracy: 0.8753 - f1_score: 0.8755 - precision_3: 0.8987 - recall_3: 0.8523 - val_loss: 0.3137 - val_accuracy: 0.8805 - val_f1_score: 0.8816 - val_precision_3: 0.9064 - val_recall_3: 0.8627
Epoch 16/25
379/379 [==============================] - 32s 82ms/step - loss: 0.3121 - accuracy: 0.8855 - f1_score: 0.8855 - precision_3: 0.9049 - recall_3: 0.8645 - val_loss: 0.3152 - val_accuracy: 0.8909 - val_f1_score: 0.8898 - val_precision_3: 0.8973 - val_recall_3: 0.8753
Epoch 17/25
379/379 [==============================] - 31s 82ms/step - loss: 0.3116 - accuracy: 0.8830 - f1_score: 0.8830 - precision_3: 0.9036 - recall_3: 0.8635 - val_loss: 0.3732 - val_accuracy: 0.8641 - val_f1_score: 0.8615 - val_precision_3: 0.8809 - val_recall_3: 0.8404
Epoch 18/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2865 - accuracy: 0.8934 - f1_score: 0.8936 - precision_3: 0.9113 - recall_3: 0.8754 - val_loss: 0.2905 - val_accuracy: 0.9042 - val_f1_score: 0.9041 - val_precision_3: 0.9132 - val_recall_3: 0.8827
Epoch 19/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2868 - accuracy: 0.8913 - f1_score: 0.8915 - precision_3: 0.9091 - recall_3: 0.8732 - val_loss: 0.2841 - val_accuracy: 0.8931 - val_f1_score: 0.8935 - val_precision_3: 0.9111 - val_recall_3: 0.8745
Epoch 20/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2677 - accuracy: 0.9007 - f1_score: 0.9008 - precision_3: 0.9162 - recall_3: 0.8830 - val_loss: 0.2580 - val_accuracy: 0.9057 - val_f1_score: 0.9059 - val_precision_3: 0.9206 - val_recall_3: 0.8946
Epoch 21/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2575 - accuracy: 0.9016 - f1_score: 0.9018 - precision_3: 0.9182 - recall_3: 0.8867 - val_loss: 0.2901 - val_accuracy: 0.9072 - val_f1_score: 0.9070 - val_precision_3: 0.9153 - val_recall_3: 0.8990
Epoch 22/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2599 - accuracy: 0.9013 - f1_score: 0.9014 - precision_3: 0.9165 - recall_3: 0.8856 - val_loss: 0.3123 - val_accuracy: 0.8976 - val_f1_score: 0.8976 - val_precision_3: 0.9128 - val_recall_3: 0.8857
Epoch 23/25
379/379 [==============================] - 32s 82ms/step - loss: 0.2479 - accuracy: 0.9057 - f1_score: 0.9058 - precision_3: 0.9215 - recall_3: 0.8930 - val_loss: 0.2545 - val_accuracy: 0.9020 - val_f1_score: 0.9020 - val_precision_3: 0.9144 - val_recall_3: 0.8886
Epoch 24/25
379/379 [==============================] - 31s 82ms/step - loss: 0.2480 - accuracy: 0.9086 - f1_score: 0.9087 - precision_3: 0.9237 - recall_3: 0.8943 - val_loss: 0.2468 - val_accuracy: 0.9079 - val_f1_score: 0.9084 - val_precision_3: 0.9260 - val_recall_3: 0.8827
Epoch 25/25
379/379 [==============================] - 31s 82ms/step - loss: 0.2272 - accuracy: 0.9137 - f1_score: 0.9137 - precision_3: 0.9259 - recall_3: 0.9027 - val_loss: 0.2982 - val_accuracy: 0.8842 - val_f1_score: 0.8853 - val_precision_3: 0.9017 - val_recall_3: 0.8716

The baseline networks started to overfit in epoch 11 so we deciced to try the model adding dropout and data augmentation, since they demonstrated more appropiate evolution of the curves. Both techniques improved considerable the results (from 0.8 to 0.9 of validation accuracy) and adjusted training and validation curves almost perfectly. These results highlight the potential of Depthwise separable convolution since with significantly fewer parameters and fewer computations compared to regular convolution, obtain comparable representational power. This results in smaller models that converge faster and are less prone to overfitting which is especially important when training small models from scratch on limited data, as in this problem.

3. Pretrained models.

Three pre-trained model architectures have been considered for applying transfer learning: VGG16, VGG19 and the best convolutional model trained from scratch in the previous section. These architectures were pre-trained on the ImageNet dataset (1.4 million labeled images and 1,000 different classes), which contains many animal classes and we can thus expect it to perform well on our problem. We will follow two different transfer learning paradigms trying to maximize the performance of those models: feature extraction and fine-tuning.

Feature extraction strategy

Feature extraction consists of using the representations learned by a previously trained model to extract interesting features from new samples. We freeze the weights in the feature extraction layer so they don't change in the new training process. These features are then run through a new classifier, which is trained from scratch.

Our classificator module will be composed by 1 hidden layer with 1024 units. We will test first the baseline models and then we will apply several modifications or regularization trying to improve the model performance if required.

Define functions

def train_model(model, train_dataset, val_dataset, epochs=20, lr=0.001, optimizer=keras.optimizers.Adam, checkpoint=None, lr_scheduler=None, early_stop=None):

  model.compile(optimizer=optimizer(learning_rate=lr),
    loss="categorical_crossentropy",
    metrics=["accuracy",keras.metrics.F1Score(),keras.metrics.Precision(),keras.metrics.Recall()]
  )

  callbacks = []
  if checkpoint is not None:
    callbacks.append(checkpoint)
  if lr_scheduler is not None:
    callbacks.append(lr_scheduler)
  if early_stop is not None:
    callbacks.append(early_stop)

  history = model.fit(train_dataset, epochs=epochs, validation_data=val_dataset, callbacks=callbacks)
  return history

def create_transfer_learning_model(cnn_model, dropout=False, data_augmentation=None):

  inputs = keras.Input(shape=(180, 180, 3))
  x = None

  if data_augmentation is not None:
    x = data_augmentation(inputs)

  # Convolutional layers
  cnn_model.trainable = False
  if x is None:
    x = cnn_model(inputs)
  else:
    x = cnn_model(x)

  x = layers.Flatten()(x)
  
  # Replace the whole classificator module since they are different problems
  x = layers.Dense(1024, activation="relu")(x)
  if (dropout):
    x = layers.Dropout(0.25)(x)
  outputs = layers.Dense(5, activation="softmax")(x)

  new_model = keras.Model(inputs, outputs)
  new_model.summary()

  return new_model

def prepare_fine_tuning(model, n_fine_tuned_layers, dropout=False):
  model.trainable = True

  # Layers to fine tune
  for layer in model.layers[:-(2+dropout+n_fine_tuned_layers)]:
    layer.trainable = False

def execute_pipeline(cnn_model, train_dataset, val_dataset, hyperparameters):

  # Set the seed to allow reproducibility
  keras.utils.set_random_seed(RANDOM_SEED)

  # Feature extraction
  model = create_transfer_learning_model(cnn_model, dropout=hyperparameters["dropout"], data_augmentation=hyperparameters["data_augmentation"])
  history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
  plot(history)

  # Fine-tuning
  if(hyperparameters["n_fine_tuned_layers"] > 0):
    prepare_fine_tuning(model,hyperparameters["n_fine_tuned_layers"],dropout=hyperparameters["dropout"])
    history = train_model(model, train_dataset, val_dataset, hyperparameters["epochs"],
                        hyperparameters["fn_lr"], hyperparameters["optimizer"],
                        hyperparameters["checkpoint"], hyperparameters["lr_scheduler"], hyperparameters["early_stop"])
    plot(history)

  return model, history

Train models

First, let's analyze what happens when we use VGG16 with our predefined classificator module.

# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

conv_base.summary()

Model: "vgg16"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_22 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 block1_conv1 (Conv2D)       (None, 180, 180, 64)      1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 180, 180, 64)      36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 90, 90, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 90, 90, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 90, 90, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 45, 45, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 45, 45, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 45, 45, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 45, 45, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 22, 22, 256)       0         
                                                                 
 block4_conv1 (Conv2D)       (None, 22, 22, 512)       1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 22, 22, 512)       2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 22, 22, 512)       2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 11, 11, 512)       0         
                                                                 
 block5_conv1 (Conv2D)       (None, 11, 11, 512)       2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 11, 11, 512)       2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 11, 11, 512)       2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 5, 5, 512)         0         
                                                                 
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

# Baseline model (VGG16)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_feature_extraction.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": keras.optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":False,
    "data_augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_3 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten (Flatten)           (None, 12800)             0         
                                                                 
 dense (Dense)               (None, 1024)              13108224  
                                                                 
 dense_1 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 13113349 (50.02 MB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
Epoch 1/50

2024-02-23 18:17:58.161910: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8904
2024-02-23 18:18:02.039417: I external/local_xla/xla/service/service.cc:168] XLA service 0x7fb9080b9ff0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-02-23 18:18:02.039453: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 3060, Compute Capability 8.6
2024-02-23 18:18:02.051939: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1708708682.157232    2520 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

379/379 [==============================] - 58s 137ms/step - loss: 0.5330 - accuracy: 0.8514 - f1_score: 0.8518 - precision: 0.8672 - recall: 0.8362 - val_loss: 0.2509 - val_accuracy: 0.8968 - val_f1_score: 0.8979 - val_precision: 0.9133 - val_recall: 0.8842
Epoch 2/50
379/379 [==============================] - 51s 133ms/step - loss: 0.1979 - accuracy: 0.9249 - f1_score: 0.9251 - precision: 0.9328 - recall: 0.9175 - val_loss: 0.2365 - val_accuracy: 0.9050 - val_f1_score: 0.9056 - val_precision: 0.9200 - val_recall: 0.8968
Epoch 3/50
379/379 [==============================] - 51s 134ms/step - loss: 0.1263 - accuracy: 0.9538 - f1_score: 0.9540 - precision: 0.9576 - recall: 0.9486 - val_loss: 0.2610 - val_accuracy: 0.9050 - val_f1_score: 0.9055 - val_precision: 0.9107 - val_recall: 0.9013
Epoch 4/50
379/379 [==============================] - 51s 133ms/step - loss: 0.0998 - accuracy: 0.9618 - f1_score: 0.9620 - precision: 0.9661 - recall: 0.9593 - val_loss: 0.2992 - val_accuracy: 0.9027 - val_f1_score: 0.9038 - val_precision: 0.9063 - val_recall: 0.8976
Epoch 5/50
379/379 [==============================] - 49s 127ms/step - loss: 0.0875 - accuracy: 0.9667 - f1_score: 0.9668 - precision: 0.9689 - recall: 0.9643 - val_loss: 0.3040 - val_accuracy: 0.9102 - val_f1_score: 0.9115 - val_precision: 0.9153 - val_recall: 0.9065
Epoch 6/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0673 - accuracy: 0.9747 - f1_score: 0.9747 - precision: 0.9764 - recall: 0.9732 - val_loss: 0.3069 - val_accuracy: 0.9117 - val_f1_score: 0.9128 - val_precision: 0.9142 - val_recall: 0.9102
Epoch 7/50
379/379 [==============================] - 30s 78ms/step - loss: 0.0464 - accuracy: 0.9842 - f1_score: 0.9842 - precision: 0.9847 - recall: 0.9833 - val_loss: 0.4038 - val_accuracy: 0.9072 - val_f1_score: 0.9076 - val_precision: 0.9071 - val_recall: 0.9065

The baseline model is a really good starting point since it provides very good results (validation accuracy: 0.9117) overcoming our previous model defined from scratch. However the training is stopped soon because the network starts to overfit in epoch 3. Let's try now to reduce the overfitting problem using dropout and data augmentation.

# VGG16 (dropout + augmentation)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_5 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 180, 180, 3)       0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten_1 (Flatten)         (None, 12800)             0         
                                                                 
 dense_2 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout (Dropout)           (None, 1024)              0         
                                                                 
 dense_3 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 13113349 (50.02 MB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 34s 83ms/step - loss: 0.6078 - accuracy: 0.8045 - f1_score: 0.8046 - precision_1: 0.8316 - recall_1: 0.7785 - val_loss: 0.3374 - val_accuracy: 0.8671 - val_f1_score: 0.8682 - val_precision_1: 0.8931 - val_recall_1: 0.8493
Epoch 2/50
379/379 [==============================] - 32s 83ms/step - loss: 0.3439 - accuracy: 0.8611 - f1_score: 0.8613 - precision_1: 0.8838 - recall_1: 0.8399 - val_loss: 0.3270 - val_accuracy: 0.8738 - val_f1_score: 0.8724 - val_precision_1: 0.8927 - val_recall_1: 0.8589
Epoch 3/50
379/379 [==============================] - 32s 83ms/step - loss: 0.3208 - accuracy: 0.8721 - f1_score: 0.8723 - precision_1: 0.8933 - recall_1: 0.8550 - val_loss: 0.2942 - val_accuracy: 0.8872 - val_f1_score: 0.8879 - val_precision_1: 0.9020 - val_recall_1: 0.8679
Epoch 4/50
379/379 [==============================] - 32s 83ms/step - loss: 0.3021 - accuracy: 0.8832 - f1_score: 0.8834 - precision_1: 0.8988 - recall_1: 0.8668 - val_loss: 0.2640 - val_accuracy: 0.8938 - val_f1_score: 0.8947 - val_precision_1: 0.9019 - val_recall_1: 0.8872
Epoch 5/50
379/379 [==============================] - 32s 83ms/step - loss: 0.2837 - accuracy: 0.8880 - f1_score: 0.8882 - precision_1: 0.9053 - recall_1: 0.8737 - val_loss: 0.3154 - val_accuracy: 0.8849 - val_f1_score: 0.8840 - val_precision_1: 0.8975 - val_recall_1: 0.8708
Epoch 6/50
379/379 [==============================] - 32s 83ms/step - loss: 0.2744 - accuracy: 0.8928 - f1_score: 0.8930 - precision_1: 0.9094 - recall_1: 0.8777 - val_loss: 0.2488 - val_accuracy: 0.8990 - val_f1_score: 0.9001 - val_precision_1: 0.9165 - val_recall_1: 0.8886
Epoch 7/50
379/379 [==============================] - 47s 123ms/step - loss: 0.2717 - accuracy: 0.8898 - f1_score: 0.8900 - precision_1: 0.9055 - recall_1: 0.8772 - val_loss: 0.2270 - val_accuracy: 0.9109 - val_f1_score: 0.9121 - val_precision_1: 0.9249 - val_recall_1: 0.9050
Epoch 8/50
379/379 [==============================] - 51s 132ms/step - loss: 0.2504 - accuracy: 0.9005 - f1_score: 0.9007 - precision_1: 0.9149 - recall_1: 0.8861 - val_loss: 0.2315 - val_accuracy: 0.9102 - val_f1_score: 0.9110 - val_precision_1: 0.9273 - val_recall_1: 0.8998
Epoch 9/50
379/379 [==============================] - 49s 127ms/step - loss: 0.2570 - accuracy: 0.8987 - f1_score: 0.8989 - precision_1: 0.9132 - recall_1: 0.8865 - val_loss: 0.2576 - val_accuracy: 0.9020 - val_f1_score: 0.9031 - val_precision_1: 0.9202 - val_recall_1: 0.8909
Epoch 10/50
379/379 [==============================] - 57s 148ms/step - loss: 0.2491 - accuracy: 0.9010 - f1_score: 0.9012 - precision_1: 0.9143 - recall_1: 0.8881 - val_loss: 0.2323 - val_accuracy: 0.9027 - val_f1_score: 0.9041 - val_precision_1: 0.9206 - val_recall_1: 0.8946
Epoch 11/50
379/379 [==============================] - 57s 149ms/step - loss: 0.2410 - accuracy: 0.9056 - f1_score: 0.9058 - precision_1: 0.9214 - recall_1: 0.8930 - val_loss: 0.2383 - val_accuracy: 0.9117 - val_f1_score: 0.9123 - val_precision_1: 0.9226 - val_recall_1: 0.9027
Epoch 12/50
379/379 [==============================] - 57s 149ms/step - loss: 0.2215 - accuracy: 0.9158 - f1_score: 0.9160 - precision_1: 0.9257 - recall_1: 0.9047 - val_loss: 0.2345 - val_accuracy: 0.9087 - val_f1_score: 0.9098 - val_precision_1: 0.9214 - val_recall_1: 0.8968

Now we can see that the overfitting problem has been highly reduced since the validation loss keeps almost horizontal. Nevertheless, we can observe some peaks derived from a high learning rate value, so let's use again the learning rate scheduler to reduce this problem and train again.

# VGG16 (dropout + augmentation + lr_scheduler)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_feature_extraction_dropout_lrScheduler.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": keras.callbacks.LearningRateScheduler(lambda epoch, lr: lr if epoch < 20 else lr * tf.math.exp(-0.1)),
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_8"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_10 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 180, 180, 3)       0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten_8 (Flatten)         (None, 12800)             0         
                                                                 
 dense_30 (Dense)            (None, 1024)              13108224  
                                                                 
 dropout_19 (Dropout)        (None, 1024)              0         
                                                                 
 dense_31 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 13113349 (50.02 MB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 61s 149ms/step - loss: 0.5908 - accuracy: 0.8070 - f1_score: 0.8074 - precision_16: 0.8342 - recall_16: 0.7823 - val_loss: 0.3672 - val_accuracy: 0.8508 - val_f1_score: 0.8482 - val_precision_16: 0.8790 - val_recall_16: 0.8411 - lr: 0.0010
Epoch 2/50
379/379 [==============================] - 52s 136ms/step - loss: 0.3448 - accuracy: 0.8639 - f1_score: 0.8642 - precision_16: 0.8840 - recall_16: 0.8430 - val_loss: 0.2543 - val_accuracy: 0.8983 - val_f1_score: 0.8990 - val_precision_16: 0.9101 - val_recall_16: 0.8864 - lr: 0.0010
Epoch 3/50
379/379 [==============================] - 52s 137ms/step - loss: 0.3147 - accuracy: 0.8787 - f1_score: 0.8789 - precision_16: 0.8960 - recall_16: 0.8614 - val_loss: 0.2696 - val_accuracy: 0.8924 - val_f1_score: 0.8934 - val_precision_16: 0.9085 - val_recall_16: 0.8775 - lr: 0.0010
Epoch 4/50
379/379 [==============================] - 52s 136ms/step - loss: 0.3012 - accuracy: 0.8798 - f1_score: 0.8800 - precision_16: 0.9004 - recall_16: 0.8621 - val_loss: 0.2841 - val_accuracy: 0.8886 - val_f1_score: 0.8899 - val_precision_16: 0.9043 - val_recall_16: 0.8768 - lr: 0.0010
Epoch 5/50
379/379 [==============================] - 52s 136ms/step - loss: 0.2767 - accuracy: 0.8946 - f1_score: 0.8948 - precision_16: 0.9108 - recall_16: 0.8789 - val_loss: 0.2263 - val_accuracy: 0.9087 - val_f1_score: 0.9100 - val_precision_16: 0.9218 - val_recall_16: 0.9013 - lr: 0.0010
Epoch 6/50
379/379 [==============================] - 52s 137ms/step - loss: 0.2722 - accuracy: 0.8934 - f1_score: 0.8937 - precision_16: 0.9096 - recall_16: 0.8783 - val_loss: 0.2228 - val_accuracy: 0.9198 - val_f1_score: 0.9209 - val_precision_16: 0.9287 - val_recall_16: 0.9087 - lr: 0.0010
Epoch 7/50
379/379 [==============================] - 53s 137ms/step - loss: 0.2657 - accuracy: 0.8940 - f1_score: 0.8942 - precision_16: 0.9090 - recall_16: 0.8775 - val_loss: 0.2550 - val_accuracy: 0.9057 - val_f1_score: 0.9065 - val_precision_16: 0.9172 - val_recall_16: 0.8968 - lr: 0.0010
Epoch 8/50
379/379 [==============================] - 53s 137ms/step - loss: 0.2532 - accuracy: 0.8985 - f1_score: 0.8987 - precision_16: 0.9131 - recall_16: 0.8856 - val_loss: 0.2466 - val_accuracy: 0.9117 - val_f1_score: 0.9126 - val_precision_16: 0.9179 - val_recall_16: 0.9042 - lr: 0.0010
Epoch 9/50
379/379 [==============================] - 52s 136ms/step - loss: 0.2494 - accuracy: 0.9030 - f1_score: 0.9033 - precision_16: 0.9167 - recall_16: 0.8898 - val_loss: 0.2262 - val_accuracy: 0.9206 - val_f1_score: 0.9212 - val_precision_16: 0.9308 - val_recall_16: 0.9087 - lr: 0.0010
Epoch 10/50
379/379 [==============================] - 52s 137ms/step - loss: 0.2509 - accuracy: 0.9013 - f1_score: 0.9014 - precision_16: 0.9157 - recall_16: 0.8896 - val_loss: 0.2364 - val_accuracy: 0.9131 - val_f1_score: 0.9141 - val_precision_16: 0.9206 - val_recall_16: 0.9035 - lr: 0.0010
Epoch 11/50
379/379 [==============================] - 52s 136ms/step - loss: 0.2322 - accuracy: 0.9085 - f1_score: 0.9088 - precision_16: 0.9217 - recall_16: 0.8972 - val_loss: 0.2573 - val_accuracy: 0.9050 - val_f1_score: 0.9046 - val_precision_16: 0.9136 - val_recall_16: 0.8946 - lr: 0.0010

Now it is the turn to test VGG19. We will analyze both the baseline model and the best regularization configuration obtained for VGG16.

# VGG19 (dropout + augmentation)
# Instantiating the VGG19 convolutional base
conv_base = keras.applications.vgg19.VGG19(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG19_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg19/vgg19_weights_tf_dim_ordering_tf_kernels_notop.h5
80134624/80134624 [==============================] - 6s 0us/step
Model: "model_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_7 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential_1 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg19 (Functional)          (None, 5, 5, 512)         20024384  
                                                                 
 flatten_2 (Flatten)         (None, 12800)             0         
                                                                 
 dense_4 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout_1 (Dropout)         (None, 1024)              0         
                                                                 
 dense_5 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 33137733 (126.41 MB)
Trainable params: 13113349 (50.02 MB)
Non-trainable params: 20024384 (76.39 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 68s 176ms/step - loss: 0.7103 - accuracy: 0.7672 - f1_score: 0.7675 - precision_2: 0.8053 - recall_2: 0.7286 - val_loss: 0.4373 - val_accuracy: 0.8070 - val_f1_score: 0.8076 - val_precision_2: 0.8237 - val_recall_2: 0.7944
Epoch 2/50
379/379 [==============================] - 67s 175ms/step - loss: 0.4113 - accuracy: 0.8333 - f1_score: 0.8338 - precision_2: 0.8587 - recall_2: 0.8021 - val_loss: 0.2962 - val_accuracy: 0.8894 - val_f1_score: 0.8907 - val_precision_2: 0.9104 - val_recall_2: 0.8671
Epoch 3/50
379/379 [==============================] - 67s 175ms/step - loss: 0.4051 - accuracy: 0.8415 - f1_score: 0.8416 - precision_2: 0.8698 - recall_2: 0.8140 - val_loss: 0.3140 - val_accuracy: 0.8775 - val_f1_score: 0.8780 - val_precision_2: 0.8995 - val_recall_2: 0.8575
Epoch 4/50
379/379 [==============================] - 67s 175ms/step - loss: 0.3796 - accuracy: 0.8492 - f1_score: 0.8495 - precision_2: 0.8739 - recall_2: 0.8216 - val_loss: 0.2989 - val_accuracy: 0.8745 - val_f1_score: 0.8753 - val_precision_2: 0.8955 - val_recall_2: 0.8589
Epoch 5/50
379/379 [==============================] - 68s 177ms/step - loss: 0.3605 - accuracy: 0.8576 - f1_score: 0.8578 - precision_2: 0.8824 - recall_2: 0.8332 - val_loss: 0.2654 - val_accuracy: 0.8976 - val_f1_score: 0.8988 - val_precision_2: 0.9165 - val_recall_2: 0.8723
Epoch 6/50
379/379 [==============================] - 68s 178ms/step - loss: 0.3454 - accuracy: 0.8613 - f1_score: 0.8615 - precision_2: 0.8876 - recall_2: 0.8395 - val_loss: 0.2528 - val_accuracy: 0.9027 - val_f1_score: 0.9040 - val_precision_2: 0.9197 - val_recall_2: 0.8924
Epoch 7/50
379/379 [==============================] - 68s 178ms/step - loss: 0.3332 - accuracy: 0.8707 - f1_score: 0.8710 - precision_2: 0.8902 - recall_2: 0.8461 - val_loss: 0.2699 - val_accuracy: 0.9020 - val_f1_score: 0.9024 - val_precision_2: 0.9228 - val_recall_2: 0.8872
Epoch 8/50
379/379 [==============================] - 68s 177ms/step - loss: 0.3383 - accuracy: 0.8646 - f1_score: 0.8648 - precision_2: 0.8893 - recall_2: 0.8434 - val_loss: 0.3264 - val_accuracy: 0.8790 - val_f1_score: 0.8814 - val_precision_2: 0.8916 - val_recall_2: 0.8731
Epoch 9/50
379/379 [==============================] - 68s 177ms/step - loss: 0.3269 - accuracy: 0.8730 - f1_score: 0.8733 - precision_2: 0.8964 - recall_2: 0.8513 - val_loss: 0.2802 - val_accuracy: 0.8924 - val_f1_score: 0.8934 - val_precision_2: 0.9039 - val_recall_2: 0.8797
Epoch 10/50
379/379 [==============================] - 65s 171ms/step - loss: 0.3224 - accuracy: 0.8711 - f1_score: 0.8713 - precision_2: 0.8938 - recall_2: 0.8516 - val_loss: 0.2680 - val_accuracy: 0.9057 - val_f1_score: 0.9063 - val_precision_2: 0.9281 - val_recall_2: 0.8812
Epoch 11/50
379/379 [==============================] - 64s 166ms/step - loss: 0.3159 - accuracy: 0.8730 - f1_score: 0.8733 - precision_2: 0.8943 - recall_2: 0.8518 - val_loss: 0.2801 - val_accuracy: 0.8953 - val_f1_score: 0.8968 - val_precision_2: 0.9079 - val_recall_2: 0.8857

# Xception (dropout + augmentation)
# Instantiating the Xception convolutional base
conv_base = keras.applications.Xception(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/Xception_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/xception/xception_weights_tf_dim_ordering_tf_kernels_notop.h5
83683744/83683744 [==============================] - 5s 0us/step
Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_5 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential (Sequential)     (None, 180, 180, 3)       0         
                                                                 
 xception (Functional)       (None, 6, 6, 2048)        20861480  
                                                                 
 flatten (Flatten)           (None, 73728)             0         
                                                                 
 dense_3 (Dense)             (None, 1024)              75498496  
                                                                 
 dropout_1 (Dropout)         (None, 1024)              0         
                                                                 
 dense_4 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 96365101 (367.60 MB)
Trainable params: 75503621 (288.02 MB)
Non-trainable params: 20861480 (79.58 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 36s 86ms/step - loss: 1.7041 - accuracy: 0.8992 - f1_score: 0.8992 - precision_3: 0.9029 - recall_3: 0.8968 - val_loss: 0.1640 - val_accuracy: 0.9540 - val_f1_score: 0.9545 - val_precision_3: 0.9588 - val_recall_3: 0.9503
Epoch 2/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1905 - accuracy: 0.9322 - f1_score: 0.9323 - precision_3: 0.9419 - recall_3: 0.9270 - val_loss: 0.1480 - val_accuracy: 0.9495 - val_f1_score: 0.9506 - val_precision_3: 0.9521 - val_recall_3: 0.9443
Epoch 3/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1877 - accuracy: 0.9349 - f1_score: 0.9350 - precision_3: 0.9435 - recall_3: 0.9288 - val_loss: 0.1292 - val_accuracy: 0.9547 - val_f1_score: 0.9554 - val_precision_3: 0.9602 - val_recall_3: 0.9495
Epoch 4/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1773 - accuracy: 0.9372 - f1_score: 0.9374 - precision_3: 0.9435 - recall_3: 0.9316 - val_loss: 0.1324 - val_accuracy: 0.9547 - val_f1_score: 0.9553 - val_precision_3: 0.9630 - val_recall_3: 0.9473
Epoch 5/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1652 - accuracy: 0.9398 - f1_score: 0.9399 - precision_3: 0.9487 - recall_3: 0.9321 - val_loss: 0.1042 - val_accuracy: 0.9607 - val_f1_score: 0.9612 - val_precision_3: 0.9627 - val_recall_3: 0.9569
Epoch 6/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1859 - accuracy: 0.9379 - f1_score: 0.9380 - precision_3: 0.9450 - recall_3: 0.9319 - val_loss: 0.1334 - val_accuracy: 0.9540 - val_f1_score: 0.9546 - val_precision_3: 0.9609 - val_recall_3: 0.9480
Epoch 7/50
379/379 [==============================] - 32s 83ms/step - loss: 0.1723 - accuracy: 0.9406 - f1_score: 0.9407 - precision_3: 0.9487 - recall_3: 0.9329 - val_loss: 0.1200 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_3: 0.9656 - val_recall_3: 0.9599
Epoch 8/50
379/379 [==============================] - 32s 83ms/step - loss: 0.1780 - accuracy: 0.9402 - f1_score: 0.9404 - precision_3: 0.9470 - recall_3: 0.9337 - val_loss: 0.1186 - val_accuracy: 0.9577 - val_f1_score: 0.9583 - val_precision_3: 0.9604 - val_recall_3: 0.9555
Epoch 9/50
379/379 [==============================] - 32s 83ms/step - loss: 0.1797 - accuracy: 0.9389 - f1_score: 0.9390 - precision_3: 0.9467 - recall_3: 0.9321 - val_loss: 0.1418 - val_accuracy: 0.9525 - val_f1_score: 0.9532 - val_precision_3: 0.9603 - val_recall_3: 0.9510
Epoch 10/50
379/379 [==============================] - 32s 84ms/step - loss: 0.1674 - accuracy: 0.9411 - f1_score: 0.9412 - precision_3: 0.9493 - recall_3: 0.9349 - val_loss: 0.1206 - val_accuracy: 0.9607 - val_f1_score: 0.9613 - val_precision_3: 0.9670 - val_recall_3: 0.9577

# InceptionV3 (dropout + augmentation)
# Instantiating the InceptionV3 convolutional base
conv_base = keras.applications.InceptionV3(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionV3_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/inception_v3/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5
87910968/87910968 [==============================] - 4s 0us/step
Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_7 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential_1 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_v3 (Functional)   (None, 4, 4, 2048)        21802784  
                                                                 
 flatten_1 (Flatten)         (None, 32768)             0         
                                                                 
 dense_5 (Dense)             (None, 1024)              33555456  
                                                                 
 dropout_2 (Dropout)         (None, 1024)              0         
                                                                 
 dense_6 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 55363365 (211.19 MB)
Trainable params: 33560581 (128.02 MB)
Non-trainable params: 21802784 (83.17 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 35s 76ms/step - loss: 2.2657 - accuracy: 0.8860 - f1_score: 0.8861 - precision_4: 0.8923 - recall_4: 0.8821 - val_loss: 0.1307 - val_accuracy: 0.9532 - val_f1_score: 0.9541 - val_precision_4: 0.9656 - val_recall_4: 0.9384
Epoch 2/50
379/379 [==============================] - 24s 61ms/step - loss: 0.2058 - accuracy: 0.9259 - f1_score: 0.9260 - precision_4: 0.9398 - recall_4: 0.9152 - val_loss: 0.1162 - val_accuracy: 0.9592 - val_f1_score: 0.9596 - val_precision_4: 0.9676 - val_recall_4: 0.9547
Epoch 3/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1870 - accuracy: 0.9325 - f1_score: 0.9326 - precision_4: 0.9436 - recall_4: 0.9223 - val_loss: 0.1179 - val_accuracy: 0.9569 - val_f1_score: 0.9576 - val_precision_4: 0.9669 - val_recall_4: 0.9532
Epoch 4/50
379/379 [==============================] - 23s 60ms/step - loss: 0.2084 - accuracy: 0.9280 - f1_score: 0.9281 - precision_4: 0.9388 - recall_4: 0.9177 - val_loss: 0.1153 - val_accuracy: 0.9592 - val_f1_score: 0.9598 - val_precision_4: 0.9669 - val_recall_4: 0.9532
Epoch 5/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1953 - accuracy: 0.9293 - f1_score: 0.9294 - precision_4: 0.9406 - recall_4: 0.9200 - val_loss: 0.1403 - val_accuracy: 0.9503 - val_f1_score: 0.9509 - val_precision_4: 0.9663 - val_recall_4: 0.9354
Epoch 6/50
379/379 [==============================] - 33s 85ms/step - loss: 0.1913 - accuracy: 0.9303 - f1_score: 0.9304 - precision_4: 0.9426 - recall_4: 0.9205 - val_loss: 0.1066 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_4: 0.9707 - val_recall_4: 0.9592
Epoch 7/50
379/379 [==============================] - 37s 96ms/step - loss: 0.1908 - accuracy: 0.9340 - f1_score: 0.9341 - precision_4: 0.9476 - recall_4: 0.9238 - val_loss: 0.1010 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_4: 0.9699 - val_recall_4: 0.9569
Epoch 8/50
379/379 [==============================] - 37s 97ms/step - loss: 0.1747 - accuracy: 0.9379 - f1_score: 0.9380 - precision_4: 0.9495 - recall_4: 0.9258 - val_loss: 0.1108 - val_accuracy: 0.9607 - val_f1_score: 0.9613 - val_precision_4: 0.9662 - val_recall_4: 0.9540
Epoch 9/50
379/379 [==============================] - 36s 94ms/step - loss: 0.1674 - accuracy: 0.9386 - f1_score: 0.9387 - precision_4: 0.9504 - recall_4: 0.9276 - val_loss: 0.1295 - val_accuracy: 0.9569 - val_f1_score: 0.9576 - val_precision_4: 0.9645 - val_recall_4: 0.9480
Epoch 10/50
379/379 [==============================] - 24s 62ms/step - loss: 0.1608 - accuracy: 0.9425 - f1_score: 0.9426 - precision_4: 0.9550 - recall_4: 0.9320 - val_loss: 0.1440 - val_accuracy: 0.9562 - val_f1_score: 0.9569 - val_precision_4: 0.9653 - val_recall_4: 0.9488
Epoch 11/50
379/379 [==============================] - 24s 62ms/step - loss: 0.1784 - accuracy: 0.9394 - f1_score: 0.9395 - precision_4: 0.9523 - recall_4: 0.9262 - val_loss: 0.1176 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_4: 0.9706 - val_recall_4: 0.9569
Epoch 12/50
379/379 [==============================] - 24s 62ms/step - loss: 0.1715 - accuracy: 0.9404 - f1_score: 0.9405 - precision_4: 0.9536 - recall_4: 0.9265 - val_loss: 0.1308 - val_accuracy: 0.9584 - val_f1_score: 0.9589 - val_precision_4: 0.9668 - val_recall_4: 0.9525

# InceptionResNetV2 (dropout + augmentation)
# Instantiating the InceptionResNetV2 convolutional base
conv_base = keras.applications.InceptionResNetV2(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionResNetV2_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_7 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential_1 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_resnet_v2 (Funct  (None, 4, 4, 1536)        54336736  
 ional)                                                          
                                                                 
 flatten_2 (Flatten)         (None, 24576)             0         
                                                                 
 dense_5 (Dense)             (None, 1024)              25166848  
                                                                 
 dropout_3 (Dropout)         (None, 1024)              0         
                                                                 
 dense_6 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 79508709 (303.30 MB)
Trainable params: 25171973 (96.02 MB)
Non-trainable params: 54336736 (207.28 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 56s 128ms/step - loss: 1.8468 - accuracy: 0.9012 - f1_score: 0.9012 - precision_4: 0.9040 - recall_4: 0.8991 - val_loss: 0.1101 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_4: 0.9693 - val_recall_4: 0.9599
Epoch 2/50
379/379 [==============================] - 45s 117ms/step - loss: 0.1937 - accuracy: 0.9332 - f1_score: 0.9333 - precision_4: 0.9421 - recall_4: 0.9252 - val_loss: 0.1062 - val_accuracy: 0.9644 - val_f1_score: 0.9650 - val_precision_4: 0.9714 - val_recall_4: 0.9584
Epoch 3/50
379/379 [==============================] - 45s 115ms/step - loss: 0.1699 - accuracy: 0.9391 - f1_score: 0.9391 - precision_4: 0.9484 - recall_4: 0.9299 - val_loss: 0.1047 - val_accuracy: 0.9629 - val_f1_score: 0.9634 - val_precision_4: 0.9678 - val_recall_4: 0.9607
Epoch 4/50
379/379 [==============================] - 42s 108ms/step - loss: 0.1632 - accuracy: 0.9441 - f1_score: 0.9442 - precision_4: 0.9534 - recall_4: 0.9351 - val_loss: 0.1163 - val_accuracy: 0.9517 - val_f1_score: 0.9525 - val_precision_4: 0.9565 - val_recall_4: 0.9465
Epoch 5/50
379/379 [==============================] - 41s 108ms/step - loss: 0.1678 - accuracy: 0.9421 - f1_score: 0.9422 - precision_4: 0.9499 - recall_4: 0.9358 - val_loss: 0.1074 - val_accuracy: 0.9673 - val_f1_score: 0.9679 - val_precision_4: 0.9701 - val_recall_4: 0.9636
Epoch 6/50
379/379 [==============================] - 45s 116ms/step - loss: 0.1691 - accuracy: 0.9381 - f1_score: 0.9382 - precision_4: 0.9458 - recall_4: 0.9314 - val_loss: 0.0951 - val_accuracy: 0.9644 - val_f1_score: 0.9649 - val_precision_4: 0.9715 - val_recall_4: 0.9607
Epoch 7/50
379/379 [==============================] - 44s 115ms/step - loss: 0.1762 - accuracy: 0.9379 - f1_score: 0.9379 - precision_4: 0.9478 - recall_4: 0.9281 - val_loss: 0.0885 - val_accuracy: 0.9710 - val_f1_score: 0.9716 - val_precision_4: 0.9738 - val_recall_4: 0.9666
Epoch 8/50
379/379 [==============================] - 42s 108ms/step - loss: 0.1723 - accuracy: 0.9409 - f1_score: 0.9409 - precision_4: 0.9499 - recall_4: 0.9319 - val_loss: 0.1110 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_4: 0.9658 - val_recall_4: 0.9644
Epoch 9/50
379/379 [==============================] - 41s 107ms/step - loss: 0.1514 - accuracy: 0.9454 - f1_score: 0.9455 - precision_4: 0.9532 - recall_4: 0.9377 - val_loss: 0.1136 - val_accuracy: 0.9644 - val_f1_score: 0.9650 - val_precision_4: 0.9665 - val_recall_4: 0.9636
Epoch 10/50
379/379 [==============================] - 41s 108ms/step - loss: 0.1773 - accuracy: 0.9395 - f1_score: 0.9395 - precision_4: 0.9511 - recall_4: 0.9307 - val_loss: 0.1146 - val_accuracy: 0.9651 - val_f1_score: 0.9656 - val_precision_4: 0.9664 - val_recall_4: 0.9607
Epoch 11/50
379/379 [==============================] - 41s 108ms/step - loss: 0.1742 - accuracy: 0.9410 - f1_score: 0.9410 - precision_4: 0.9500 - recall_4: 0.9327 - val_loss: 0.1013 - val_accuracy: 0.9666 - val_f1_score: 0.9671 - val_precision_4: 0.9708 - val_recall_4: 0.9621
Epoch 12/50
379/379 [==============================] - 41s 107ms/step - loss: 0.1597 - accuracy: 0.9475 - f1_score: 0.9475 - precision_4: 0.9570 - recall_4: 0.9388 - val_loss: 0.0955 - val_accuracy: 0.9659 - val_f1_score: 0.9663 - val_precision_4: 0.9723 - val_recall_4: 0.9636

EXPERIMENT: Just with experimental purposes we will try to use the best model trained from scratch in the previous section and we will apply feature extraction. Since it was trained with the same dataset the convolutional block already knows the existing patterns on our data so let's see what happens if we just change the classification block of the model.

# Best scratch model
conv_base = keras.models.load_model(
    OUT_MODELS_FOLDER + "/cnn_reg_lr0.0005_dropout_aug_lrScheduler.keras")

checkpoint_filename = "/VGG16_feature_extraction_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 0,  # apart from the Denses, starting from the bottom of the network

    "dropout":False,
    "data_augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_5 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 model_1 (Functional)        (None, 5)                 3095749   
                                                                 
 flatten_4 (Flatten)         (None, 5)                 0         
                                                                 
 dense_12 (Dense)            (None, 1024)              6144      
                                                                 
 dense_13 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 3107018 (11.85 MB)
Trainable params: 11269 (44.02 KB)
Non-trainable params: 3095749 (11.81 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 21s 51ms/step - loss: 0.3052 - accuracy: 0.9276 - f1_score: 0.9278 - precision_8: 0.9420 - recall_8: 0.8529 - val_loss: 0.4110 - val_accuracy: 0.8716 - val_f1_score: 0.8736 - val_precision_8: 0.8816 - val_recall_8: 0.8679
Epoch 2/50
379/379 [==============================] - 20s 52ms/step - loss: 0.2055 - accuracy: 0.9341 - f1_score: 0.9343 - precision_8: 0.9416 - recall_8: 0.9274 - val_loss: 0.4118 - val_accuracy: 0.8686 - val_f1_score: 0.8708 - val_precision_8: 0.8789 - val_recall_8: 0.8679
Epoch 3/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1981 - accuracy: 0.9310 - f1_score: 0.9312 - precision_8: 0.9385 - recall_8: 0.9238 - val_loss: 0.4096 - val_accuracy: 0.8693 - val_f1_score: 0.8716 - val_precision_8: 0.8794 - val_recall_8: 0.8664
Epoch 4/50
379/379 [==============================] - 20s 52ms/step - loss: 0.2004 - accuracy: 0.9335 - f1_score: 0.9337 - precision_8: 0.9408 - recall_8: 0.9273 - val_loss: 0.4239 - val_accuracy: 0.8701 - val_f1_score: 0.8722 - val_precision_8: 0.8787 - val_recall_8: 0.8656
Epoch 5/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1906 - accuracy: 0.9354 - f1_score: 0.9356 - precision_8: 0.9430 - recall_8: 0.9282 - val_loss: 0.4114 - val_accuracy: 0.8708 - val_f1_score: 0.8729 - val_precision_8: 0.8821 - val_recall_8: 0.8664
Epoch 6/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1905 - accuracy: 0.9363 - f1_score: 0.9365 - precision_8: 0.9431 - recall_8: 0.9288 - val_loss: 0.4123 - val_accuracy: 0.8701 - val_f1_score: 0.8721 - val_precision_8: 0.8801 - val_recall_8: 0.8664
Epoch 7/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1889 - accuracy: 0.9343 - f1_score: 0.9345 - precision_8: 0.9433 - recall_8: 0.9267 - val_loss: 0.4059 - val_accuracy: 0.8731 - val_f1_score: 0.8749 - val_precision_8: 0.8854 - val_recall_8: 0.8664
Epoch 8/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1952 - accuracy: 0.9339 - f1_score: 0.9341 - precision_8: 0.9426 - recall_8: 0.9250 - val_loss: 0.4040 - val_accuracy: 0.8708 - val_f1_score: 0.8730 - val_precision_8: 0.8846 - val_recall_8: 0.8649
Epoch 9/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1936 - accuracy: 0.9327 - f1_score: 0.9328 - precision_8: 0.9420 - recall_8: 0.9234 - val_loss: 0.4010 - val_accuracy: 0.8738 - val_f1_score: 0.8756 - val_precision_8: 0.8828 - val_recall_8: 0.8612
Epoch 10/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1939 - accuracy: 0.9326 - f1_score: 0.9328 - precision_8: 0.9439 - recall_8: 0.9229 - val_loss: 0.4003 - val_accuracy: 0.8731 - val_f1_score: 0.8749 - val_precision_8: 0.8853 - val_recall_8: 0.8656
Epoch 11/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1871 - accuracy: 0.9348 - f1_score: 0.9350 - precision_8: 0.9450 - recall_8: 0.9260 - val_loss: 0.4132 - val_accuracy: 0.8745 - val_f1_score: 0.8763 - val_precision_8: 0.8845 - val_recall_8: 0.8641
Epoch 12/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1886 - accuracy: 0.9341 - f1_score: 0.9343 - precision_8: 0.9415 - recall_8: 0.9257 - val_loss: 0.4124 - val_accuracy: 0.8738 - val_f1_score: 0.8756 - val_precision_8: 0.8852 - val_recall_8: 0.8641
Epoch 13/50
379/379 [==============================] - 21s 54ms/step - loss: 0.1866 - accuracy: 0.9337 - f1_score: 0.9338 - precision_8: 0.9444 - recall_8: 0.9239 - val_loss: 0.4070 - val_accuracy: 0.8716 - val_f1_score: 0.8735 - val_precision_8: 0.8826 - val_recall_8: 0.8649
Epoch 14/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1920 - accuracy: 0.9321 - f1_score: 0.9322 - precision_8: 0.9412 - recall_8: 0.9211 - val_loss: 0.4014 - val_accuracy: 0.8745 - val_f1_score: 0.8763 - val_precision_8: 0.8871 - val_recall_8: 0.8634
Epoch 15/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1880 - accuracy: 0.9337 - f1_score: 0.9338 - precision_8: 0.9448 - recall_8: 0.9252 - val_loss: 0.3984 - val_accuracy: 0.8753 - val_f1_score: 0.8772 - val_precision_8: 0.8833 - val_recall_8: 0.8656
Epoch 16/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1872 - accuracy: 0.9343 - f1_score: 0.9345 - precision_8: 0.9452 - recall_8: 0.9247 - val_loss: 0.4075 - val_accuracy: 0.8738 - val_f1_score: 0.8756 - val_precision_8: 0.8833 - val_recall_8: 0.8656
Epoch 17/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1882 - accuracy: 0.9340 - f1_score: 0.9341 - precision_8: 0.9443 - recall_8: 0.9247 - val_loss: 0.3990 - val_accuracy: 0.8731 - val_f1_score: 0.8748 - val_precision_8: 0.8866 - val_recall_8: 0.8649
Epoch 18/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1919 - accuracy: 0.9331 - f1_score: 0.9333 - precision_8: 0.9443 - recall_8: 0.9232 - val_loss: 0.4060 - val_accuracy: 0.8731 - val_f1_score: 0.8749 - val_precision_8: 0.8865 - val_recall_8: 0.8641
Epoch 19/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1833 - accuracy: 0.9340 - f1_score: 0.9342 - precision_8: 0.9454 - recall_8: 0.9240 - val_loss: 0.4092 - val_accuracy: 0.8723 - val_f1_score: 0.8742 - val_precision_8: 0.8844 - val_recall_8: 0.8634
Epoch 20/50
379/379 [==============================] - 20s 52ms/step - loss: 0.1852 - accuracy: 0.9344 - f1_score: 0.9346 - precision_8: 0.9454 - recall_8: 0.9247 - val_loss: 0.4081 - val_accuracy: 0.8731 - val_f1_score: 0.8749 - val_precision_8: 0.8844 - val_recall_8: 0.8634

It is interesting to see that the loss curve remains constant across epochs, except for the first two epochs, due to the feature extraction block had already learned features specific to the dataset during the initial training (recall that the convolutional base is by far the most expensive part of the pipeline). However, it is always a good practice to experiment with different approaches and evaluate their performance empirically. Sometimes, even if the initial features learned during pre-training overlap with the dataset, fine-tuning certain layers or adapting the network's architecture can still lead to improvements in performance, but this is not the case as the performance remains the same.

Fine-tuning

Fine-tuning consists of unfreezing a few of the top layers of a frozen model base used for feature extraction. Then jointly train both these top layers and the added part (the fully connected classifier). It slightly adjusts the more abstract features reused in order to make them more relevant for the problem at hand.

Same as before, our classificator module will be composed by 1 hidden layer with 1024 units. The steps for fine-tuning a network are as follows:

1. Add our custom network on top of an already-trained base network.
2. Freeze the base network.
3. Train the part we added.

• This are the same steps as in feature extraction:
  • We need to train the classifier part before fine-tuning.
  • Otherwise the error signal during training will be too large and it will destroy the features already learned in the layer we are going to fine-tune.

4. Unfreeze some layers in the base network.
5. Jointly train both those layers and the classifier added (again).

We will test first the baseline models and then we will apply several modifications or regularization trying to improve the model performance if required.

Let's start doing a fine-tuning of the last block of the convolutional model. It is important to recall that fine-tuning requires a low learning rate.

# VGG16 fine-tuning the last layer (dropout)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_finetuning_dropout.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": None,
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_4 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten_1 (Flatten)         (None, 12800)             0         
                                                                 
 dense_2 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout_1 (Dropout)         (None, 1024)              0         
                                                                 
 dense_3 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 20192773 (77.03 MB)
Non-trainable params: 7635264 (29.13 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 33s 81ms/step - loss: 0.4237 - accuracy: 0.8335 - f1_score: 0.8337 - precision_1: 0.8906 - recall_1: 0.7823 - val_loss: 0.2152 - val_accuracy: 0.9154 - val_f1_score: 0.9167 - val_precision_1: 0.9298 - val_recall_1: 0.9050
Epoch 2/50
379/379 [==============================] - 31s 79ms/step - loss: 0.1768 - accuracy: 0.9323 - f1_score: 0.9325 - precision_1: 0.9430 - recall_1: 0.9230 - val_loss: 0.1894 - val_accuracy: 0.9228 - val_f1_score: 0.9236 - val_precision_1: 0.9346 - val_recall_1: 0.9124
Epoch 3/50
379/379 [==============================] - 31s 79ms/step - loss: 0.1125 - accuracy: 0.9600 - f1_score: 0.9601 - precision_1: 0.9663 - recall_1: 0.9533 - val_loss: 0.1697 - val_accuracy: 0.9376 - val_f1_score: 0.9384 - val_precision_1: 0.9437 - val_recall_1: 0.9324
Epoch 4/50
379/379 [==============================] - 31s 80ms/step - loss: 0.0720 - accuracy: 0.9751 - f1_score: 0.9751 - precision_1: 0.9788 - recall_1: 0.9717 - val_loss: 0.1639 - val_accuracy: 0.9376 - val_f1_score: 0.9384 - val_precision_1: 0.9437 - val_recall_1: 0.9324
Epoch 5/50
379/379 [==============================] - 31s 79ms/step - loss: 0.0455 - accuracy: 0.9874 - f1_score: 0.9874 - precision_1: 0.9905 - recall_1: 0.9854 - val_loss: 0.1747 - val_accuracy: 0.9399 - val_f1_score: 0.9406 - val_precision_1: 0.9446 - val_recall_1: 0.9362
Epoch 6/50
379/379 [==============================] - 31s 80ms/step - loss: 0.0268 - accuracy: 0.9935 - f1_score: 0.9935 - precision_1: 0.9948 - recall_1: 0.9925 - val_loss: 0.1730 - val_accuracy: 0.9414 - val_f1_score: 0.9422 - val_precision_1: 0.9461 - val_recall_1: 0.9376
Epoch 7/50
379/379 [==============================] - 31s 80ms/step - loss: 0.0202 - accuracy: 0.9958 - f1_score: 0.9958 - precision_1: 0.9964 - recall_1: 0.9949 - val_loss: 0.1826 - val_accuracy: 0.9376 - val_f1_score: 0.9384 - val_precision_1: 0.9424 - val_recall_1: 0.9347
Epoch 8/50
379/379 [==============================] - 30s 79ms/step - loss: 0.0132 - accuracy: 0.9974 - f1_score: 0.9974 - precision_1: 0.9979 - recall_1: 0.9974 - val_loss: 0.1984 - val_accuracy: 0.9362 - val_f1_score: 0.9368 - val_precision_1: 0.9382 - val_recall_1: 0.9354
Epoch 9/50
379/379 [==============================] - 31s 79ms/step - loss: 0.0095 - accuracy: 0.9987 - f1_score: 0.9987 - precision_1: 0.9988 - recall_1: 0.9986 - val_loss: 0.1987 - val_accuracy: 0.9384 - val_f1_score: 0.9391 - val_precision_1: 0.9411 - val_recall_1: 0.9376

At a first glance, we can see that the validation results are really good, with a validation accuracy of 0.9414. In addition, the model does not seem to be suffering from an important overfitting, since the slope of the loss curve is almost horizontal, but let's try what happens when we apply regularization. We will use dropout together with data augmentation and we will test 2 different options of fine-tuning (taking either 1 or 2 convolutional blocks).

# VGG16 fine-tuning the last layer (dropout + data augmentation)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_finetuning_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_11 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_3 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten_4 (Flatten)         (None, 12800)             0         
                                                                 
 dense_8 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout_4 (Dropout)         (None, 1024)              0         
                                                                 
 dense_9 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 20192773 (77.03 MB)
Non-trainable params: 7635264 (29.13 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 34s 85ms/step - loss: 0.5036 - accuracy: 0.7992 - f1_score: 0.7995 - precision_4: 0.8652 - recall_4: 0.7352 - val_loss: 0.2342 - val_accuracy: 0.9117 - val_f1_score: 0.9126 - val_precision_4: 0.9317 - val_recall_4: 0.9013
Epoch 2/50
379/379 [==============================] - 33s 85ms/step - loss: 0.2702 - accuracy: 0.8954 - f1_score: 0.8957 - precision_4: 0.9118 - recall_4: 0.8781 - val_loss: 0.1953 - val_accuracy: 0.9198 - val_f1_score: 0.9207 - val_precision_4: 0.9308 - val_recall_4: 0.9087
Epoch 3/50
379/379 [==============================] - 32s 82ms/step - loss: 0.2162 - accuracy: 0.9139 - f1_score: 0.9141 - precision_4: 0.9266 - recall_4: 0.9037 - val_loss: 0.1702 - val_accuracy: 0.9317 - val_f1_score: 0.9324 - val_precision_4: 0.9430 - val_recall_4: 0.9213
Epoch 4/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1818 - accuracy: 0.9304 - f1_score: 0.9306 - precision_4: 0.9386 - recall_4: 0.9209 - val_loss: 0.1597 - val_accuracy: 0.9362 - val_f1_score: 0.9370 - val_precision_4: 0.9464 - val_recall_4: 0.9310
Epoch 5/50
379/379 [==============================] - 31s 81ms/step - loss: 0.1688 - accuracy: 0.9397 - f1_score: 0.9399 - precision_4: 0.9473 - recall_4: 0.9305 - val_loss: 0.1649 - val_accuracy: 0.9391 - val_f1_score: 0.9399 - val_precision_4: 0.9451 - val_recall_4: 0.9324
Epoch 6/50
379/379 [==============================] - 31s 81ms/step - loss: 0.1453 - accuracy: 0.9453 - f1_score: 0.9454 - precision_4: 0.9532 - recall_4: 0.9378 - val_loss: 0.1618 - val_accuracy: 0.9399 - val_f1_score: 0.9408 - val_precision_4: 0.9473 - val_recall_4: 0.9339
Epoch 7/50
379/379 [==============================] - 31s 81ms/step - loss: 0.1305 - accuracy: 0.9504 - f1_score: 0.9505 - precision_4: 0.9570 - recall_4: 0.9443 - val_loss: 0.1527 - val_accuracy: 0.9391 - val_f1_score: 0.9397 - val_precision_4: 0.9481 - val_recall_4: 0.9362
Epoch 8/50
379/379 [==============================] - 32s 82ms/step - loss: 0.1143 - accuracy: 0.9567 - f1_score: 0.9568 - precision_4: 0.9622 - recall_4: 0.9522 - val_loss: 0.1531 - val_accuracy: 0.9399 - val_f1_score: 0.9405 - val_precision_4: 0.9445 - val_recall_4: 0.9354
Epoch 9/50
379/379 [==============================] - 31s 81ms/step - loss: 0.1075 - accuracy: 0.9599 - f1_score: 0.9600 - precision_4: 0.9648 - recall_4: 0.9558 - val_loss: 0.1373 - val_accuracy: 0.9473 - val_f1_score: 0.9480 - val_precision_4: 0.9571 - val_recall_4: 0.9443
Epoch 10/50
379/379 [==============================] - 32s 81ms/step - loss: 0.0977 - accuracy: 0.9653 - f1_score: 0.9653 - precision_4: 0.9702 - recall_4: 0.9612 - val_loss: 0.1381 - val_accuracy: 0.9480 - val_f1_score: 0.9486 - val_precision_4: 0.9521 - val_recall_4: 0.9436
Epoch 11/50
379/379 [==============================] - 31s 82ms/step - loss: 0.0915 - accuracy: 0.9672 - f1_score: 0.9672 - precision_4: 0.9716 - recall_4: 0.9638 - val_loss: 0.1625 - val_accuracy: 0.9406 - val_f1_score: 0.9418 - val_precision_4: 0.9461 - val_recall_4: 0.9391
Epoch 12/50
379/379 [==============================] - 31s 81ms/step - loss: 0.0817 - accuracy: 0.9706 - f1_score: 0.9706 - precision_4: 0.9748 - recall_4: 0.9680 - val_loss: 0.1276 - val_accuracy: 0.9547 - val_f1_score: 0.9553 - val_precision_4: 0.9581 - val_recall_4: 0.9503
Epoch 13/50
379/379 [==============================] - 31s 81ms/step - loss: 0.0758 - accuracy: 0.9713 - f1_score: 0.9714 - precision_4: 0.9750 - recall_4: 0.9687 - val_loss: 0.1293 - val_accuracy: 0.9547 - val_f1_score: 0.9552 - val_precision_4: 0.9589 - val_recall_4: 0.9517
Epoch 14/50
379/379 [==============================] - 31s 81ms/step - loss: 0.0643 - accuracy: 0.9763 - f1_score: 0.9764 - precision_4: 0.9795 - recall_4: 0.9742 - val_loss: 0.1308 - val_accuracy: 0.9584 - val_f1_score: 0.9591 - val_precision_4: 0.9597 - val_recall_4: 0.9555
Epoch 15/50
379/379 [==============================] - 31s 81ms/step - loss: 0.0656 - accuracy: 0.9769 - f1_score: 0.9770 - precision_4: 0.9796 - recall_4: 0.9751 - val_loss: 0.1474 - val_accuracy: 0.9555 - val_f1_score: 0.9559 - val_precision_4: 0.9596 - val_recall_4: 0.9532
Epoch 16/50
379/379 [==============================] - 31s 81ms/step - loss: 0.0640 - accuracy: 0.9765 - f1_score: 0.9765 - precision_4: 0.9795 - recall_4: 0.9751 - val_loss: 0.1463 - val_accuracy: 0.9532 - val_f1_score: 0.9537 - val_precision_4: 0.9552 - val_recall_4: 0.9503
Epoch 17/50
379/379 [==============================] - 32s 84ms/step - loss: 0.0576 - accuracy: 0.9787 - f1_score: 0.9788 - precision_4: 0.9808 - recall_4: 0.9772 - val_loss: 0.1529 - val_accuracy: 0.9473 - val_f1_score: 0.9479 - val_precision_4: 0.9500 - val_recall_4: 0.9451

# VGG16 fine-tuning the two last layers (dropout + data augmentation)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg16.VGG16(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG16_finetuning2_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 8,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_9 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential_2 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg16 (Functional)          (None, 5, 5, 512)         14714688  
                                                                 
 flatten_3 (Flatten)         (None, 12800)             0         
                                                                 
 dense_6 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout_2 (Dropout)         (None, 1024)              0         
                                                                 
 dense_7 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 27828037 (106.16 MB)
Trainable params: 26092549 (99.54 MB)
Non-trainable params: 1735488 (6.62 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 86s 203ms/step - loss: 0.4356 - accuracy: 0.8268 - f1_score: 0.8272 - precision_3: 0.8779 - recall_3: 0.7764 - val_loss: 0.1951 - val_accuracy: 0.9250 - val_f1_score: 0.9258 - val_precision_3: 0.9393 - val_recall_3: 0.9079
Epoch 2/50
379/379 [==============================] - 73s 190ms/step - loss: 0.2022 - accuracy: 0.9226 - f1_score: 0.9227 - precision_3: 0.9345 - recall_3: 0.9137 - val_loss: 0.1520 - val_accuracy: 0.9428 - val_f1_score: 0.9435 - val_precision_3: 0.9502 - val_recall_3: 0.9339
Epoch 3/50
379/379 [==============================] - 73s 190ms/step - loss: 0.1654 - accuracy: 0.9383 - f1_score: 0.9385 - precision_3: 0.9456 - recall_3: 0.9314 - val_loss: 0.1422 - val_accuracy: 0.9465 - val_f1_score: 0.9470 - val_precision_3: 0.9535 - val_recall_3: 0.9428
Epoch 4/50
379/379 [==============================] - 73s 190ms/step - loss: 0.1291 - accuracy: 0.9520 - f1_score: 0.9521 - precision_3: 0.9574 - recall_3: 0.9462 - val_loss: 0.1124 - val_accuracy: 0.9569 - val_f1_score: 0.9575 - val_precision_3: 0.9611 - val_recall_3: 0.9540
Epoch 5/50
379/379 [==============================] - 73s 190ms/step - loss: 0.1217 - accuracy: 0.9544 - f1_score: 0.9545 - precision_3: 0.9600 - recall_3: 0.9485 - val_loss: 0.1187 - val_accuracy: 0.9599 - val_f1_score: 0.9605 - val_precision_3: 0.9656 - val_recall_3: 0.9584
Epoch 6/50
379/379 [==============================] - 73s 190ms/step - loss: 0.0936 - accuracy: 0.9659 - f1_score: 0.9660 - precision_3: 0.9692 - recall_3: 0.9621 - val_loss: 0.1590 - val_accuracy: 0.9510 - val_f1_score: 0.9519 - val_precision_3: 0.9530 - val_recall_3: 0.9488
Epoch 7/50
379/379 [==============================] - 73s 190ms/step - loss: 0.0822 - accuracy: 0.9709 - f1_score: 0.9709 - precision_3: 0.9743 - recall_3: 0.9670 - val_loss: 0.1322 - val_accuracy: 0.9555 - val_f1_score: 0.9560 - val_precision_3: 0.9568 - val_recall_3: 0.9540
Epoch 8/50
379/379 [==============================] - 73s 190ms/step - loss: 0.0714 - accuracy: 0.9737 - f1_score: 0.9737 - precision_3: 0.9761 - recall_3: 0.9711 - val_loss: 0.1127 - val_accuracy: 0.9607 - val_f1_score: 0.9611 - val_precision_3: 0.9642 - val_recall_3: 0.9599
Epoch 9/50
379/379 [==============================] - 73s 190ms/step - loss: 0.0635 - accuracy: 0.9772 - f1_score: 0.9772 - precision_3: 0.9799 - recall_3: 0.9750 - val_loss: 0.1469 - val_accuracy: 0.9517 - val_f1_score: 0.9525 - val_precision_3: 0.9558 - val_recall_3: 0.9480

From the plots of the 1-block fine-tuning we can see how the application of regularization achieves to delay the overfiting emergence until epoch 12 and improve the validation accuracy (from 0.9414 to 0.9584).

Regarding 2-block fine-tuning we achieve to overcome these results and obtain an impressive performance with a validation accuracy of 0.9607, the best found until now.

Now it is the turn to analyze VGG19. We will apply regularization techniques as they have helped to improve the performance of VGG16 and we will test again 2 different options of fine-tuning (taking either 1 or 2 convolutional blocks).

# VGG19 fine-tuning the last layer (dropout + data augmentation)
# Instantiating the VGG16 convolutional base
conv_base = keras.applications.vgg19.VGG19(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG19_finetuning_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_11 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_3 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg19 (Functional)          (None, 5, 5, 512)         20024384  
                                                                 
 flatten_4 (Flatten)         (None, 12800)             0         
                                                                 
 dense_8 (Dense)             (None, 1024)              13108224  
                                                                 
 dropout_3 (Dropout)         (None, 1024)              0         
                                                                 
 dense_9 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 33137733 (126.41 MB)
Trainable params: 20192773 (77.03 MB)
Non-trainable params: 12944960 (49.38 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 66s 168ms/step - loss: 0.5655 - accuracy: 0.7705 - f1_score: 0.7713 - precision_4: 0.8493 - recall_4: 0.6991 - val_loss: 0.2732 - val_accuracy: 0.8990 - val_f1_score: 0.9002 - val_precision_4: 0.9160 - val_recall_4: 0.8745
Epoch 2/50
379/379 [==============================] - 64s 167ms/step - loss: 0.3324 - accuracy: 0.8695 - f1_score: 0.8698 - precision_4: 0.8925 - recall_4: 0.8468 - val_loss: 0.2504 - val_accuracy: 0.8998 - val_f1_score: 0.9007 - val_precision_4: 0.9234 - val_recall_4: 0.8857
Epoch 3/50
379/379 [==============================] - 64s 167ms/step - loss: 0.2672 - accuracy: 0.8964 - f1_score: 0.8967 - precision_4: 0.9123 - recall_4: 0.8810 - val_loss: 0.2134 - val_accuracy: 0.9206 - val_f1_score: 0.9215 - val_precision_4: 0.9350 - val_recall_4: 0.9072
Epoch 4/50
379/379 [==============================] - 64s 168ms/step - loss: 0.2410 - accuracy: 0.9069 - f1_score: 0.9071 - precision_4: 0.9208 - recall_4: 0.8942 - val_loss: 0.2002 - val_accuracy: 0.9258 - val_f1_score: 0.9264 - val_precision_4: 0.9397 - val_recall_4: 0.9139
Epoch 5/50
379/379 [==============================] - 64s 167ms/step - loss: 0.2125 - accuracy: 0.9204 - f1_score: 0.9207 - precision_4: 0.9319 - recall_4: 0.9083 - val_loss: 0.2096 - val_accuracy: 0.9213 - val_f1_score: 0.9223 - val_precision_4: 0.9346 - val_recall_4: 0.9013
Epoch 6/50
379/379 [==============================] - 64s 167ms/step - loss: 0.1946 - accuracy: 0.9266 - f1_score: 0.9268 - precision_4: 0.9361 - recall_4: 0.9175 - val_loss: 0.1708 - val_accuracy: 0.9354 - val_f1_score: 0.9361 - val_precision_4: 0.9454 - val_recall_4: 0.9258
Epoch 7/50
379/379 [==============================] - 64s 167ms/step - loss: 0.1797 - accuracy: 0.9306 - f1_score: 0.9307 - precision_4: 0.9401 - recall_4: 0.9198 - val_loss: 0.1714 - val_accuracy: 0.9391 - val_f1_score: 0.9400 - val_precision_4: 0.9509 - val_recall_4: 0.9347
Epoch 8/50
379/379 [==============================] - 64s 167ms/step - loss: 0.1602 - accuracy: 0.9415 - f1_score: 0.9416 - precision_4: 0.9482 - recall_4: 0.9320 - val_loss: 0.2146 - val_accuracy: 0.9243 - val_f1_score: 0.9259 - val_precision_4: 0.9284 - val_recall_4: 0.9139
Epoch 9/50
379/379 [==============================] - 64s 168ms/step - loss: 0.1519 - accuracy: 0.9425 - f1_score: 0.9427 - precision_4: 0.9502 - recall_4: 0.9339 - val_loss: 0.1928 - val_accuracy: 0.9250 - val_f1_score: 0.9259 - val_precision_4: 0.9358 - val_recall_4: 0.9206
Epoch 10/50
379/379 [==============================] - 64s 167ms/step - loss: 0.1357 - accuracy: 0.9476 - f1_score: 0.9478 - precision_4: 0.9545 - recall_4: 0.9418 - val_loss: 0.1700 - val_accuracy: 0.9406 - val_f1_score: 0.9416 - val_precision_4: 0.9486 - val_recall_4: 0.9310
Epoch 11/50
379/379 [==============================] - 61s 159ms/step - loss: 0.1238 - accuracy: 0.9527 - f1_score: 0.9528 - precision_4: 0.9591 - recall_4: 0.9476 - val_loss: 0.1827 - val_accuracy: 0.9362 - val_f1_score: 0.9369 - val_precision_4: 0.9421 - val_recall_4: 0.9302
Epoch 12/50
379/379 [==============================] - 36s 93ms/step - loss: 0.1166 - accuracy: 0.9543 - f1_score: 0.9544 - precision_4: 0.9603 - recall_4: 0.9504 - val_loss: 0.1595 - val_accuracy: 0.9436 - val_f1_score: 0.9444 - val_precision_4: 0.9517 - val_recall_4: 0.9362
Epoch 13/50
379/379 [==============================] - 36s 93ms/step - loss: 0.1088 - accuracy: 0.9598 - f1_score: 0.9599 - precision_4: 0.9651 - recall_4: 0.9544 - val_loss: 0.1691 - val_accuracy: 0.9458 - val_f1_score: 0.9465 - val_precision_4: 0.9491 - val_recall_4: 0.9421
Epoch 14/50
379/379 [==============================] - 36s 93ms/step - loss: 0.1005 - accuracy: 0.9617 - f1_score: 0.9617 - precision_4: 0.9660 - recall_4: 0.9578 - val_loss: 0.1689 - val_accuracy: 0.9451 - val_f1_score: 0.9457 - val_precision_4: 0.9505 - val_recall_4: 0.9406
Epoch 15/50
379/379 [==============================] - 36s 93ms/step - loss: 0.1012 - accuracy: 0.9620 - f1_score: 0.9621 - precision_4: 0.9658 - recall_4: 0.9581 - val_loss: 0.1749 - val_accuracy: 0.9369 - val_f1_score: 0.9380 - val_precision_4: 0.9430 - val_recall_4: 0.9332
Epoch 16/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0967 - accuracy: 0.9641 - f1_score: 0.9642 - precision_4: 0.9686 - recall_4: 0.9601 - val_loss: 0.1562 - val_accuracy: 0.9451 - val_f1_score: 0.9459 - val_precision_4: 0.9490 - val_recall_4: 0.9391
Epoch 17/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0838 - accuracy: 0.9691 - f1_score: 0.9691 - precision_4: 0.9723 - recall_4: 0.9653 - val_loss: 0.1617 - val_accuracy: 0.9465 - val_f1_score: 0.9471 - val_precision_4: 0.9526 - val_recall_4: 0.9406
Epoch 18/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0823 - accuracy: 0.9706 - f1_score: 0.9706 - precision_4: 0.9739 - recall_4: 0.9673 - val_loss: 0.2008 - val_accuracy: 0.9317 - val_f1_score: 0.9324 - val_precision_4: 0.9348 - val_recall_4: 0.9265
Epoch 19/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0754 - accuracy: 0.9719 - f1_score: 0.9719 - precision_4: 0.9744 - recall_4: 0.9692 - val_loss: 0.1526 - val_accuracy: 0.9451 - val_f1_score: 0.9459 - val_precision_4: 0.9504 - val_recall_4: 0.9391
Epoch 20/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0759 - accuracy: 0.9734 - f1_score: 0.9734 - precision_4: 0.9764 - recall_4: 0.9699 - val_loss: 0.1867 - val_accuracy: 0.9376 - val_f1_score: 0.9382 - val_precision_4: 0.9410 - val_recall_4: 0.9354
Epoch 21/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0671 - accuracy: 0.9766 - f1_score: 0.9766 - precision_4: 0.9783 - recall_4: 0.9745 - val_loss: 0.1509 - val_accuracy: 0.9510 - val_f1_score: 0.9517 - val_precision_4: 0.9537 - val_recall_4: 0.9480
Epoch 22/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0615 - accuracy: 0.9772 - f1_score: 0.9772 - precision_4: 0.9795 - recall_4: 0.9753 - val_loss: 0.1516 - val_accuracy: 0.9540 - val_f1_score: 0.9545 - val_precision_4: 0.9567 - val_recall_4: 0.9517
Epoch 23/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0607 - accuracy: 0.9772 - f1_score: 0.9773 - precision_4: 0.9794 - recall_4: 0.9753 - val_loss: 0.1849 - val_accuracy: 0.9376 - val_f1_score: 0.9385 - val_precision_4: 0.9438 - val_recall_4: 0.9354
Epoch 24/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0545 - accuracy: 0.9800 - f1_score: 0.9800 - precision_4: 0.9827 - recall_4: 0.9784 - val_loss: 0.1826 - val_accuracy: 0.9391 - val_f1_score: 0.9403 - val_precision_4: 0.9425 - val_recall_4: 0.9369
Epoch 25/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0543 - accuracy: 0.9818 - f1_score: 0.9818 - precision_4: 0.9832 - recall_4: 0.9795 - val_loss: 0.1777 - val_accuracy: 0.9458 - val_f1_score: 0.9466 - val_precision_4: 0.9491 - val_recall_4: 0.9421
Epoch 26/50
379/379 [==============================] - 36s 93ms/step - loss: 0.0516 - accuracy: 0.9821 - f1_score: 0.9821 - precision_4: 0.9835 - recall_4: 0.9805 - val_loss: 0.2159 - val_accuracy: 0.9295 - val_f1_score: 0.9311 - val_precision_4: 0.9321 - val_recall_4: 0.9272

# VGG19 fine-tuning the two last layers (dropout + data augmentation)
# Instantiating the VGG19 convolutional base
conv_base = keras.applications.vgg19.VGG19(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/VGG19_finetuning2_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 8,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_13 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_4 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 vgg19 (Functional)          (None, 5, 5, 512)         20024384  
                                                                 
 flatten_5 (Flatten)         (None, 12800)             0         
                                                                 
 dense_10 (Dense)            (None, 1024)              13108224  
                                                                 
 dropout_4 (Dropout)         (None, 1024)              0         
                                                                 
 dense_11 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 33137733 (126.41 MB)
Trainable params: 27272197 (104.04 MB)
Non-trainable params: 5865536 (22.38 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 45s 111ms/step - loss: 0.4731 - accuracy: 0.8153 - f1_score: 0.8158 - precision_5: 0.8698 - recall_5: 0.7595 - val_loss: 0.2610 - val_accuracy: 0.8938 - val_f1_score: 0.8956 - val_precision_5: 0.9131 - val_recall_5: 0.8738
Epoch 2/50
379/379 [==============================] - 43s 111ms/step - loss: 0.2448 - accuracy: 0.9058 - f1_score: 0.9061 - precision_5: 0.9182 - recall_5: 0.8932 - val_loss: 0.1699 - val_accuracy: 0.9465 - val_f1_score: 0.9472 - val_precision_5: 0.9539 - val_recall_5: 0.9369
Epoch 3/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1940 - accuracy: 0.9236 - f1_score: 0.9237 - precision_5: 0.9354 - recall_5: 0.9162 - val_loss: 0.1868 - val_accuracy: 0.9258 - val_f1_score: 0.9273 - val_precision_5: 0.9351 - val_recall_5: 0.9198
Epoch 4/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1611 - accuracy: 0.9387 - f1_score: 0.9389 - precision_5: 0.9481 - recall_5: 0.9316 - val_loss: 0.1453 - val_accuracy: 0.9443 - val_f1_score: 0.9447 - val_precision_5: 0.9534 - val_recall_5: 0.9414
Epoch 5/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1341 - accuracy: 0.9492 - f1_score: 0.9493 - precision_5: 0.9560 - recall_5: 0.9443 - val_loss: 0.1149 - val_accuracy: 0.9562 - val_f1_score: 0.9567 - val_precision_5: 0.9633 - val_recall_5: 0.9540
Epoch 6/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1180 - accuracy: 0.9544 - f1_score: 0.9545 - precision_5: 0.9590 - recall_5: 0.9504 - val_loss: 0.1355 - val_accuracy: 0.9517 - val_f1_score: 0.9523 - val_precision_5: 0.9574 - val_recall_5: 0.9503
Epoch 7/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1026 - accuracy: 0.9627 - f1_score: 0.9628 - precision_5: 0.9670 - recall_5: 0.9593 - val_loss: 0.1223 - val_accuracy: 0.9592 - val_f1_score: 0.9597 - val_precision_5: 0.9641 - val_recall_5: 0.9569
Epoch 8/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0940 - accuracy: 0.9647 - f1_score: 0.9648 - precision_5: 0.9688 - recall_5: 0.9612 - val_loss: 0.1441 - val_accuracy: 0.9495 - val_f1_score: 0.9503 - val_precision_5: 0.9523 - val_recall_5: 0.9480
Epoch 9/50
379/379 [==============================] - 54s 142ms/step - loss: 0.0849 - accuracy: 0.9681 - f1_score: 0.9681 - precision_5: 0.9710 - recall_5: 0.9650 - val_loss: 0.1201 - val_accuracy: 0.9614 - val_f1_score: 0.9619 - val_precision_5: 0.9621 - val_recall_5: 0.9599
Epoch 10/50
379/379 [==============================] - 78s 203ms/step - loss: 0.0730 - accuracy: 0.9747 - f1_score: 0.9747 - precision_5: 0.9771 - recall_5: 0.9721 - val_loss: 0.1144 - val_accuracy: 0.9599 - val_f1_score: 0.9604 - val_precision_5: 0.9634 - val_recall_5: 0.9584
Epoch 11/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0675 - accuracy: 0.9758 - f1_score: 0.9758 - precision_5: 0.9781 - recall_5: 0.9727 - val_loss: 0.1101 - val_accuracy: 0.9696 - val_f1_score: 0.9700 - val_precision_5: 0.9709 - val_recall_5: 0.9659
Epoch 12/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0580 - accuracy: 0.9792 - f1_score: 0.9793 - precision_5: 0.9818 - recall_5: 0.9769 - val_loss: 0.1345 - val_accuracy: 0.9532 - val_f1_score: 0.9540 - val_precision_5: 0.9581 - val_recall_5: 0.9503
Epoch 13/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0562 - accuracy: 0.9798 - f1_score: 0.9798 - precision_5: 0.9816 - recall_5: 0.9779 - val_loss: 0.1763 - val_accuracy: 0.9443 - val_f1_score: 0.9453 - val_precision_5: 0.9492 - val_recall_5: 0.9436
Epoch 14/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0503 - accuracy: 0.9803 - f1_score: 0.9803 - precision_5: 0.9824 - recall_5: 0.9789 - val_loss: 0.1258 - val_accuracy: 0.9621 - val_f1_score: 0.9626 - val_precision_5: 0.9621 - val_recall_5: 0.9607
Epoch 15/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0468 - accuracy: 0.9826 - f1_score: 0.9826 - precision_5: 0.9844 - recall_5: 0.9816 - val_loss: 0.1513 - val_accuracy: 0.9569 - val_f1_score: 0.9574 - val_precision_5: 0.9604 - val_recall_5: 0.9540
Epoch 16/50
379/379 [==============================] - 78s 204ms/step - loss: 0.0442 - accuracy: 0.9846 - f1_score: 0.9846 - precision_5: 0.9859 - recall_5: 0.9830 - val_loss: 0.1239 - val_accuracy: 0.9614 - val_f1_score: 0.9618 - val_precision_5: 0.9627 - val_recall_5: 0.9592

Again fine-tuning 2 convolutional blocks provided the best results with a validation accuracy of 0.9696. Thus the best model obtained using the fine-tuning technique was VGG19 fine-tuning 2 convolutional blocks with dropout and data augmentation.

# Xception fine-tuning the last layer (dropout + data augmentation)
# Instantiating the Xception convolutional base
conv_base = keras.applications.Xception(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/Xception_finetuning_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_11 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_3 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 xception (Functional)       (None, 6, 6, 2048)        20861480  
                                                                 
 flatten_4 (Flatten)         (None, 73728)             0         
                                                                 
 dense_9 (Dense)             (None, 1024)              75498496  
                                                                 
 dropout_5 (Dropout)         (None, 1024)              0         
                                                                 
 dense_10 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 96365101 (367.60 MB)
Trainable params: 78667269 (300.09 MB)
Non-trainable params: 17697832 (67.51 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 46s 104ms/step - loss: 0.2356 - accuracy: 0.9143 - f1_score: 0.9144 - precision_6: 0.9409 - recall_6: 0.8907 - val_loss: 0.1220 - val_accuracy: 0.9562 - val_f1_score: 0.9568 - val_precision_6: 0.9589 - val_recall_6: 0.9532
Epoch 2/50
379/379 [==============================] - 36s 92ms/step - loss: 0.1531 - accuracy: 0.9416 - f1_score: 0.9418 - precision_6: 0.9504 - recall_6: 0.9340 - val_loss: 0.1089 - val_accuracy: 0.9651 - val_f1_score: 0.9656 - val_precision_6: 0.9679 - val_recall_6: 0.9621
Epoch 3/50
379/379 [==============================] - 36s 94ms/step - loss: 0.1385 - accuracy: 0.9499 - f1_score: 0.9501 - precision_6: 0.9572 - recall_6: 0.9435 - val_loss: 0.1080 - val_accuracy: 0.9592 - val_f1_score: 0.9598 - val_precision_6: 0.9627 - val_recall_6: 0.9577
Epoch 4/50
379/379 [==============================] - 33s 85ms/step - loss: 0.1205 - accuracy: 0.9543 - f1_score: 0.9544 - precision_6: 0.9595 - recall_6: 0.9480 - val_loss: 0.1119 - val_accuracy: 0.9621 - val_f1_score: 0.9627 - val_precision_6: 0.9634 - val_recall_6: 0.9584
Epoch 5/50
379/379 [==============================] - 35s 92ms/step - loss: 0.1135 - accuracy: 0.9577 - f1_score: 0.9578 - precision_6: 0.9620 - recall_6: 0.9532 - val_loss: 0.0995 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_6: 0.9672 - val_recall_6: 0.9636
Epoch 6/50
379/379 [==============================] - 36s 92ms/step - loss: 0.1036 - accuracy: 0.9626 - f1_score: 0.9626 - precision_6: 0.9660 - recall_6: 0.9572 - val_loss: 0.0989 - val_accuracy: 0.9644 - val_f1_score: 0.9649 - val_precision_6: 0.9657 - val_recall_6: 0.9621
Epoch 7/50
379/379 [==============================] - 36s 92ms/step - loss: 0.1026 - accuracy: 0.9612 - f1_score: 0.9613 - precision_6: 0.9652 - recall_6: 0.9570 - val_loss: 0.0957 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_6: 0.9658 - val_recall_6: 0.9644
Epoch 8/50
379/379 [==============================] - 33s 86ms/step - loss: 0.0966 - accuracy: 0.9640 - f1_score: 0.9641 - precision_6: 0.9678 - recall_6: 0.9596 - val_loss: 0.0961 - val_accuracy: 0.9629 - val_f1_score: 0.9634 - val_precision_6: 0.9650 - val_recall_6: 0.9614
Epoch 9/50
379/379 [==============================] - 33s 85ms/step - loss: 0.0820 - accuracy: 0.9693 - f1_score: 0.9694 - precision_6: 0.9722 - recall_6: 0.9664 - val_loss: 0.1054 - val_accuracy: 0.9636 - val_f1_score: 0.9641 - val_precision_6: 0.9650 - val_recall_6: 0.9614
Epoch 10/50
379/379 [==============================] - 33s 85ms/step - loss: 0.0802 - accuracy: 0.9729 - f1_score: 0.9729 - precision_6: 0.9750 - recall_6: 0.9697 - val_loss: 0.1033 - val_accuracy: 0.9659 - val_f1_score: 0.9664 - val_precision_6: 0.9672 - val_recall_6: 0.9644
Epoch 11/50
379/379 [==============================] - 33s 85ms/step - loss: 0.0743 - accuracy: 0.9716 - f1_score: 0.9717 - precision_6: 0.9746 - recall_6: 0.9692 - val_loss: 0.1043 - val_accuracy: 0.9659 - val_f1_score: 0.9664 - val_precision_6: 0.9680 - val_recall_6: 0.9644
Epoch 12/50
379/379 [==============================] - 33s 86ms/step - loss: 0.0701 - accuracy: 0.9739 - f1_score: 0.9739 - precision_6: 0.9761 - recall_6: 0.9715 - val_loss: 0.1086 - val_accuracy: 0.9659 - val_f1_score: 0.9663 - val_precision_6: 0.9658 - val_recall_6: 0.9644

# Xception fine-tuning the two last layers (dropout + data augmentation)
# Instantiating the Xception convolutional base
conv_base = keras.applications.Xception(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/Xception_finetuning2_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 8,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_7"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_13 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_4 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 xception (Functional)       (None, 6, 6, 2048)        20861480  
                                                                 
 flatten_4 (Flatten)         (None, 73728)             0         
                                                                 
 dense_11 (Dense)            (None, 1024)              75498496  
                                                                 
 dropout_5 (Dropout)         (None, 1024)              0         
                                                                 
 dense_12 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 96365101 (367.60 MB)
Trainable params: 80254469 (306.15 MB)
Non-trainable params: 16110632 (61.46 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 63s 155ms/step - loss: 0.2677 - accuracy: 0.9037 - f1_score: 0.9039 - precision_7: 0.9331 - recall_7: 0.8779 - val_loss: 0.1024 - val_accuracy: 0.9696 - val_f1_score: 0.9701 - val_precision_7: 0.9723 - val_recall_7: 0.9659
Epoch 2/50
379/379 [==============================] - 59s 154ms/step - loss: 0.1626 - accuracy: 0.9406 - f1_score: 0.9408 - precision_7: 0.9478 - recall_7: 0.9320 - val_loss: 0.0900 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_7: 0.9746 - val_recall_7: 0.9673
Epoch 3/50
379/379 [==============================] - 39s 102ms/step - loss: 0.1439 - accuracy: 0.9448 - f1_score: 0.9449 - precision_7: 0.9514 - recall_7: 0.9378 - val_loss: 0.0868 - val_accuracy: 0.9681 - val_f1_score: 0.9685 - val_precision_7: 0.9716 - val_recall_7: 0.9644
Epoch 4/50
379/379 [==============================] - 33s 86ms/step - loss: 0.1309 - accuracy: 0.9504 - f1_score: 0.9505 - precision_7: 0.9578 - recall_7: 0.9443 - val_loss: 0.0880 - val_accuracy: 0.9651 - val_f1_score: 0.9658 - val_precision_7: 0.9694 - val_recall_7: 0.9651
Epoch 5/50
379/379 [==============================] - 35s 91ms/step - loss: 0.1183 - accuracy: 0.9560 - f1_score: 0.9561 - precision_7: 0.9617 - recall_7: 0.9509 - val_loss: 0.0840 - val_accuracy: 0.9703 - val_f1_score: 0.9708 - val_precision_7: 0.9739 - val_recall_7: 0.9688
Epoch 6/50
379/379 [==============================] - 34s 87ms/step - loss: 0.1080 - accuracy: 0.9615 - f1_score: 0.9616 - precision_7: 0.9662 - recall_7: 0.9560 - val_loss: 0.0840 - val_accuracy: 0.9688 - val_f1_score: 0.9694 - val_precision_7: 0.9702 - val_recall_7: 0.9659
Epoch 7/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0958 - accuracy: 0.9661 - f1_score: 0.9662 - precision_7: 0.9699 - recall_7: 0.9616 - val_loss: 0.0840 - val_accuracy: 0.9710 - val_f1_score: 0.9715 - val_precision_7: 0.9731 - val_recall_7: 0.9673
Epoch 8/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0874 - accuracy: 0.9665 - f1_score: 0.9666 - precision_7: 0.9710 - recall_7: 0.9642 - val_loss: 0.0938 - val_accuracy: 0.9681 - val_f1_score: 0.9686 - val_precision_7: 0.9702 - val_recall_7: 0.9659
Epoch 9/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0897 - accuracy: 0.9678 - f1_score: 0.9679 - precision_7: 0.9716 - recall_7: 0.9640 - val_loss: 0.0838 - val_accuracy: 0.9703 - val_f1_score: 0.9707 - val_precision_7: 0.9717 - val_recall_7: 0.9688
Epoch 10/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0795 - accuracy: 0.9717 - f1_score: 0.9718 - precision_7: 0.9749 - recall_7: 0.9691 - val_loss: 0.0889 - val_accuracy: 0.9644 - val_f1_score: 0.9649 - val_precision_7: 0.9665 - val_recall_7: 0.9636
Epoch 11/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0744 - accuracy: 0.9729 - f1_score: 0.9729 - precision_7: 0.9753 - recall_7: 0.9700 - val_loss: 0.0834 - val_accuracy: 0.9703 - val_f1_score: 0.9708 - val_precision_7: 0.9710 - val_recall_7: 0.9681
Epoch 12/50
379/379 [==============================] - 34s 88ms/step - loss: 0.0655 - accuracy: 0.9764 - f1_score: 0.9765 - precision_7: 0.9782 - recall_7: 0.9747 - val_loss: 0.0857 - val_accuracy: 0.9681 - val_f1_score: 0.9686 - val_precision_7: 0.9688 - val_recall_7: 0.9666
Epoch 13/50
379/379 [==============================] - 34s 89ms/step - loss: 0.0614 - accuracy: 0.9772 - f1_score: 0.9772 - precision_7: 0.9790 - recall_7: 0.9756 - val_loss: 0.0887 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_7: 0.9695 - val_recall_7: 0.9688
Epoch 14/50
379/379 [==============================] - 34s 89ms/step - loss: 0.0629 - accuracy: 0.9765 - f1_score: 0.9766 - precision_7: 0.9793 - recall_7: 0.9752 - val_loss: 0.0931 - val_accuracy: 0.9673 - val_f1_score: 0.9679 - val_precision_7: 0.9673 - val_recall_7: 0.9651
Epoch 15/50
379/379 [==============================] - 34s 89ms/step - loss: 0.0574 - accuracy: 0.9803 - f1_score: 0.9803 - precision_7: 0.9816 - recall_7: 0.9791 - val_loss: 0.0925 - val_accuracy: 0.9688 - val_f1_score: 0.9694 - val_precision_7: 0.9688 - val_recall_7: 0.9681
Epoch 16/50
379/379 [==============================] - 33s 86ms/step - loss: 0.0555 - accuracy: 0.9800 - f1_score: 0.9801 - precision_7: 0.9817 - recall_7: 0.9790 - val_loss: 0.0910 - val_accuracy: 0.9666 - val_f1_score: 0.9671 - val_precision_7: 0.9680 - val_recall_7: 0.9666

# InceptionV3 fine-tuning the last layer (dropout + data augmentation)
# Instantiating the InceptionV3 convolutional base
conv_base = keras.applications.InceptionV3(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionV3_finetuning_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_8"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_15 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_5 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_v3 (Functional)   (None, 4, 4, 2048)        21802784  
                                                                 
 flatten_5 (Flatten)         (None, 32768)             0         
                                                                 
 dense_13 (Dense)            (None, 1024)              33555456  
                                                                 
 dropout_6 (Dropout)         (None, 1024)              0         
                                                                 
 dense_14 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 55363365 (211.19 MB)
Trainable params: 33560581 (128.02 MB)
Non-trainable params: 21802784 (83.17 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 27s 62ms/step - loss: 0.3301 - accuracy: 0.8883 - f1_score: 0.8885 - precision_8: 0.9018 - recall_8: 0.8771 - val_loss: 0.1245 - val_accuracy: 0.9540 - val_f1_score: 0.9548 - val_precision_8: 0.9587 - val_recall_8: 0.9488
Epoch 2/50
259/379 [===================>..........] - ETA: 6s - loss: 0.1970 - accuracy: 0.9313 - f1_score: 0.9316 - precision_8: 0.9391 - recall_8: 0.9262

379/379 [==============================] - 23s 60ms/step - loss: 0.1888 - accuracy: 0.9334 - f1_score: 0.9335 - precision_8: 0.9401 - recall_8: 0.9278 - val_loss: 0.1032 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_8: 0.9700 - val_recall_8: 0.9592
Epoch 3/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1645 - accuracy: 0.9381 - f1_score: 0.9382 - precision_8: 0.9463 - recall_8: 0.9336 - val_loss: 0.1030 - val_accuracy: 0.9607 - val_f1_score: 0.9613 - val_precision_8: 0.9641 - val_recall_8: 0.9562
Epoch 4/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1432 - accuracy: 0.9466 - f1_score: 0.9467 - precision_8: 0.9510 - recall_8: 0.9421 - val_loss: 0.0996 - val_accuracy: 0.9599 - val_f1_score: 0.9606 - val_precision_8: 0.9634 - val_recall_8: 0.9577
Epoch 5/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1369 - accuracy: 0.9481 - f1_score: 0.9483 - precision_8: 0.9541 - recall_8: 0.9439 - val_loss: 0.1148 - val_accuracy: 0.9607 - val_f1_score: 0.9613 - val_precision_8: 0.9626 - val_recall_8: 0.9562
Epoch 6/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1267 - accuracy: 0.9514 - f1_score: 0.9515 - precision_8: 0.9560 - recall_8: 0.9476 - val_loss: 0.1170 - val_accuracy: 0.9584 - val_f1_score: 0.9591 - val_precision_8: 0.9633 - val_recall_8: 0.9555
Epoch 7/50
379/379 [==============================] - 23s 59ms/step - loss: 0.1229 - accuracy: 0.9528 - f1_score: 0.9529 - precision_8: 0.9573 - recall_8: 0.9492 - val_loss: 0.0988 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_8: 0.9686 - val_recall_8: 0.9621
Epoch 8/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1146 - accuracy: 0.9565 - f1_score: 0.9566 - precision_8: 0.9613 - recall_8: 0.9532 - val_loss: 0.0953 - val_accuracy: 0.9621 - val_f1_score: 0.9628 - val_precision_8: 0.9664 - val_recall_8: 0.9614
Epoch 9/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1043 - accuracy: 0.9596 - f1_score: 0.9597 - precision_8: 0.9634 - recall_8: 0.9557 - val_loss: 0.0900 - val_accuracy: 0.9696 - val_f1_score: 0.9700 - val_precision_8: 0.9724 - val_recall_8: 0.9673
Epoch 10/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1030 - accuracy: 0.9607 - f1_score: 0.9607 - precision_8: 0.9656 - recall_8: 0.9575 - val_loss: 0.1024 - val_accuracy: 0.9659 - val_f1_score: 0.9664 - val_precision_8: 0.9687 - val_recall_8: 0.9644
Epoch 11/50
379/379 [==============================] - 23s 59ms/step - loss: 0.0952 - accuracy: 0.9638 - f1_score: 0.9639 - precision_8: 0.9670 - recall_8: 0.9603 - val_loss: 0.0961 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_8: 0.9710 - val_recall_8: 0.9681
Epoch 12/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0908 - accuracy: 0.9649 - f1_score: 0.9649 - precision_8: 0.9681 - recall_8: 0.9623 - val_loss: 0.1407 - val_accuracy: 0.9532 - val_f1_score: 0.9543 - val_precision_8: 0.9574 - val_recall_8: 0.9510
Epoch 13/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0892 - accuracy: 0.9664 - f1_score: 0.9665 - precision_8: 0.9697 - recall_8: 0.9627 - val_loss: 0.1190 - val_accuracy: 0.9584 - val_f1_score: 0.9591 - val_precision_8: 0.9613 - val_recall_8: 0.9577
Epoch 14/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0884 - accuracy: 0.9678 - f1_score: 0.9679 - precision_8: 0.9700 - recall_8: 0.9658 - val_loss: 0.1059 - val_accuracy: 0.9659 - val_f1_score: 0.9665 - val_precision_8: 0.9680 - val_recall_8: 0.9659

# InceptionV3 fine-tuning the two last layers (dropout + data augmentation)
# Instantiating the InceptionV3 convolutional base
conv_base = keras.applications.InceptionV3(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionV3_finetuning2_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 8,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_9"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_17 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_6 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_v3 (Functional)   (None, 4, 4, 2048)        21802784  
                                                                 
 flatten_6 (Flatten)         (None, 32768)             0         
                                                                 
 dense_15 (Dense)            (None, 1024)              33555456  
                                                                 
 dropout_7 (Dropout)         (None, 1024)              0         
                                                                 
 dense_16 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 55363365 (211.19 MB)
Trainable params: 33560773 (128.02 MB)
Non-trainable params: 21802592 (83.17 MB)
_________________________________________________________________
Epoch 1/50
110/379 [=======>......................] - ETA: 14s - loss: 0.5180 - accuracy: 0.8230 - f1_score: 0.8239 - precision_9: 0.8549 - recall_9: 0.8054

379/379 [==============================] - 27s 63ms/step - loss: 0.3280 - accuracy: 0.8855 - f1_score: 0.8856 - precision_9: 0.9012 - recall_9: 0.8738 - val_loss: 0.1609 - val_accuracy: 0.9421 - val_f1_score: 0.9428 - val_precision_9: 0.9518 - val_recall_9: 0.9384
Epoch 2/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1852 - accuracy: 0.9307 - f1_score: 0.9309 - precision_9: 0.9379 - recall_9: 0.9245 - val_loss: 0.1165 - val_accuracy: 0.9592 - val_f1_score: 0.9598 - val_precision_9: 0.9626 - val_recall_9: 0.9555
Epoch 3/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1612 - accuracy: 0.9390 - f1_score: 0.9391 - precision_9: 0.9447 - recall_9: 0.9341 - val_loss: 0.1060 - val_accuracy: 0.9644 - val_f1_score: 0.9649 - val_precision_9: 0.9708 - val_recall_9: 0.9621
Epoch 4/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1480 - accuracy: 0.9446 - f1_score: 0.9447 - precision_9: 0.9517 - recall_9: 0.9392 - val_loss: 0.0990 - val_accuracy: 0.9636 - val_f1_score: 0.9641 - val_precision_9: 0.9686 - val_recall_9: 0.9614
Epoch 5/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1333 - accuracy: 0.9493 - f1_score: 0.9494 - precision_9: 0.9541 - recall_9: 0.9444 - val_loss: 0.0987 - val_accuracy: 0.9621 - val_f1_score: 0.9628 - val_precision_9: 0.9678 - val_recall_9: 0.9584
Epoch 6/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1267 - accuracy: 0.9531 - f1_score: 0.9532 - precision_9: 0.9586 - recall_9: 0.9485 - val_loss: 0.1012 - val_accuracy: 0.9621 - val_f1_score: 0.9627 - val_precision_9: 0.9657 - val_recall_9: 0.9614
Epoch 7/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1182 - accuracy: 0.9552 - f1_score: 0.9553 - precision_9: 0.9605 - recall_9: 0.9509 - val_loss: 0.0993 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_9: 0.9657 - val_recall_9: 0.9621
Epoch 8/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1172 - accuracy: 0.9560 - f1_score: 0.9561 - precision_9: 0.9606 - recall_9: 0.9524 - val_loss: 0.0989 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_9: 0.9649 - val_recall_9: 0.9599
Epoch 9/50
379/379 [==============================] - 24s 61ms/step - loss: 0.1063 - accuracy: 0.9604 - f1_score: 0.9605 - precision_9: 0.9642 - recall_9: 0.9560 - val_loss: 0.0982 - val_accuracy: 0.9621 - val_f1_score: 0.9627 - val_precision_9: 0.9678 - val_recall_9: 0.9584
Epoch 10/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1060 - accuracy: 0.9592 - f1_score: 0.9593 - precision_9: 0.9630 - recall_9: 0.9555 - val_loss: 0.0895 - val_accuracy: 0.9696 - val_f1_score: 0.9700 - val_precision_9: 0.9731 - val_recall_9: 0.9666
Epoch 11/50
379/379 [==============================] - 23s 60ms/step - loss: 0.1036 - accuracy: 0.9607 - f1_score: 0.9608 - precision_9: 0.9649 - recall_9: 0.9577 - val_loss: 0.1017 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_9: 0.9665 - val_recall_9: 0.9629
Epoch 12/50
379/379 [==============================] - 24s 61ms/step - loss: 0.0907 - accuracy: 0.9658 - f1_score: 0.9658 - precision_9: 0.9693 - recall_9: 0.9630 - val_loss: 0.0947 - val_accuracy: 0.9666 - val_f1_score: 0.9672 - val_precision_9: 0.9687 - val_recall_9: 0.9651
Epoch 13/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0937 - accuracy: 0.9646 - f1_score: 0.9647 - precision_9: 0.9680 - recall_9: 0.9617 - val_loss: 0.0943 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_9: 0.9658 - val_recall_9: 0.9636
Epoch 14/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0913 - accuracy: 0.9636 - f1_score: 0.9636 - precision_9: 0.9676 - recall_9: 0.9617 - val_loss: 0.0963 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_9: 0.9724 - val_recall_9: 0.9673
Epoch 15/50
379/379 [==============================] - 23s 60ms/step - loss: 0.0828 - accuracy: 0.9706 - f1_score: 0.9706 - precision_9: 0.9735 - recall_9: 0.9676 - val_loss: 0.1001 - val_accuracy: 0.9636 - val_f1_score: 0.9642 - val_precision_9: 0.9671 - val_recall_9: 0.9607

# InceptionResNetV2 fine-tuning the last layer (dropout + data augmentation)
# Instantiating the InceptionResNetV2 convolutional base
conv_base = keras.applications.InceptionResNetV2(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionResNetV2_finetuning_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 4,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": None
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_10"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_19 (InputLayer)       [(None, 180, 180, 3)]     0         
                                                                 
 sequential_7 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_resnet_v2 (Funct  (None, 4, 4, 1536)        54336736  
 ional)                                                          
                                                                 
 flatten_7 (Flatten)         (None, 24576)             0         
                                                                 
 dense_17 (Dense)            (None, 1024)              25166848  
                                                                 
 dropout_8 (Dropout)         (None, 1024)              0         
                                                                 
 dense_18 (Dense)            (None, 5)                 5125      
                                                                 
=================================================================
Total params: 79508709 (303.30 MB)
Trainable params: 28368389 (108.22 MB)
Non-trainable params: 51140320 (195.08 MB)
_________________________________________________________________
Epoch 1/50

379/379 [==============================] - 59s 127ms/step - loss: 0.2591 - accuracy: 0.9026 - f1_score: 0.9026 - precision_10: 0.9310 - recall_10: 0.8791 - val_loss: 0.1030 - val_accuracy: 0.9651 - val_f1_score: 0.9656 - val_precision_10: 0.9743 - val_recall_10: 0.9577
Epoch 2/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1645 - accuracy: 0.9395 - f1_score: 0.9395 - precision_10: 0.9496 - recall_10: 0.9300 - val_loss: 0.0989 - val_accuracy: 0.9644 - val_f1_score: 0.9650 - val_precision_10: 0.9722 - val_recall_10: 0.9599
Epoch 3/50
379/379 [==============================] - 44s 114ms/step - loss: 0.1446 - accuracy: 0.9481 - f1_score: 0.9482 - precision_10: 0.9549 - recall_10: 0.9408 - val_loss: 0.0956 - val_accuracy: 0.9651 - val_f1_score: 0.9657 - val_precision_10: 0.9686 - val_recall_10: 0.9621
Epoch 4/50
379/379 [==============================] - 43s 111ms/step - loss: 0.1238 - accuracy: 0.9532 - f1_score: 0.9532 - precision_10: 0.9601 - recall_10: 0.9471 - val_loss: 0.0896 - val_accuracy: 0.9673 - val_f1_score: 0.9679 - val_precision_10: 0.9701 - val_recall_10: 0.9636
Epoch 5/50
379/379 [==============================] - 42s 110ms/step - loss: 0.1152 - accuracy: 0.9567 - f1_score: 0.9568 - precision_10: 0.9622 - recall_10: 0.9517 - val_loss: 0.0869 - val_accuracy: 0.9725 - val_f1_score: 0.9730 - val_precision_10: 0.9768 - val_recall_10: 0.9703
Epoch 6/50
379/379 [==============================] - 43s 112ms/step - loss: 0.1044 - accuracy: 0.9619 - f1_score: 0.9620 - precision_10: 0.9664 - recall_10: 0.9576 - val_loss: 0.0869 - val_accuracy: 0.9651 - val_f1_score: 0.9656 - val_precision_10: 0.9723 - val_recall_10: 0.9629
Epoch 7/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0967 - accuracy: 0.9647 - f1_score: 0.9648 - precision_10: 0.9695 - recall_10: 0.9600 - val_loss: 0.0877 - val_accuracy: 0.9659 - val_f1_score: 0.9664 - val_precision_10: 0.9709 - val_recall_10: 0.9644
Epoch 8/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0948 - accuracy: 0.9650 - f1_score: 0.9650 - precision_10: 0.9699 - recall_10: 0.9621 - val_loss: 0.0933 - val_accuracy: 0.9688 - val_f1_score: 0.9694 - val_precision_10: 0.9709 - val_recall_10: 0.9673
Epoch 9/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0901 - accuracy: 0.9657 - f1_score: 0.9658 - precision_10: 0.9689 - recall_10: 0.9628 - val_loss: 0.0849 - val_accuracy: 0.9696 - val_f1_score: 0.9700 - val_precision_10: 0.9723 - val_recall_10: 0.9651
Epoch 10/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0806 - accuracy: 0.9701 - f1_score: 0.9701 - precision_10: 0.9730 - recall_10: 0.9674 - val_loss: 0.0893 - val_accuracy: 0.9644 - val_f1_score: 0.9649 - val_precision_10: 0.9658 - val_recall_10: 0.9636
Epoch 11/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0758 - accuracy: 0.9721 - f1_score: 0.9722 - precision_10: 0.9751 - recall_10: 0.9693 - val_loss: 0.0798 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_10: 0.9732 - val_recall_10: 0.9710
Epoch 12/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0722 - accuracy: 0.9750 - f1_score: 0.9751 - precision_10: 0.9772 - recall_10: 0.9722 - val_loss: 0.0830 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_10: 0.9701 - val_recall_10: 0.9651
Epoch 13/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0667 - accuracy: 0.9757 - f1_score: 0.9757 - precision_10: 0.9780 - recall_10: 0.9736 - val_loss: 0.0779 - val_accuracy: 0.9710 - val_f1_score: 0.9714 - val_precision_10: 0.9732 - val_recall_10: 0.9710
Epoch 14/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0655 - accuracy: 0.9753 - f1_score: 0.9754 - precision_10: 0.9779 - recall_10: 0.9729 - val_loss: 0.0783 - val_accuracy: 0.9673 - val_f1_score: 0.9678 - val_precision_10: 0.9695 - val_recall_10: 0.9666
Epoch 15/50
379/379 [==============================] - 43s 111ms/step - loss: 0.0616 - accuracy: 0.9777 - f1_score: 0.9778 - precision_10: 0.9799 - recall_10: 0.9750 - val_loss: 0.0894 - val_accuracy: 0.9673 - val_f1_score: 0.9677 - val_precision_10: 0.9694 - val_recall_10: 0.9659
Epoch 16/50
379/379 [==============================] - 42s 110ms/step - loss: 0.0574 - accuracy: 0.9788 - f1_score: 0.9789 - precision_10: 0.9807 - recall_10: 0.9773 - val_loss: 0.0855 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_10: 0.9695 - val_recall_10: 0.9681
Epoch 17/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0555 - accuracy: 0.9796 - f1_score: 0.9797 - precision_10: 0.9819 - recall_10: 0.9782 - val_loss: 0.0806 - val_accuracy: 0.9703 - val_f1_score: 0.9707 - val_precision_10: 0.9732 - val_recall_10: 0.9696
Epoch 18/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0533 - accuracy: 0.9816 - f1_score: 0.9817 - precision_10: 0.9829 - recall_10: 0.9801 - val_loss: 0.0794 - val_accuracy: 0.9710 - val_f1_score: 0.9714 - val_precision_10: 0.9710 - val_recall_10: 0.9688

# InceptionResNetV2 fine-tuning the two last layers (dropout + data augmentation)
# Instantiating the InceptionResNetV2 convolutional base
conv_base = keras.applications.InceptionResNetV2(
    weights="imagenet",
    include_top=False,
    input_shape=(IMG_HEIGHT, IMG_WIDTH, 3))

checkpoint_filename = "/InceptionResNetV2_finetuning2_dropout_da.keras"
hyperparameters = {
    "epochs": 50,
    "lr": 0.001,
    "fn_lr": 1e-5, # We’ll use a very low learning rate when fine-tuning our model.
    "optimizer": optimizers.Adam,

    "n_fine_tuned_layers": 8,  # apart from the Denses, starting from the bottom of the network

    "dropout":True,
    "data_augmentation": keras.Sequential(
        [
            layers.RandomFlip("horizontal"),
            layers.RandomRotation(0.1),
            layers.RandomZoom(0.2),
        ]
    ),
    "lr_scheduler": None,
    "early_stop": keras.callbacks.EarlyStopping(monitor='val_loss', patience=5),
    "checkpoint": keras.callbacks.ModelCheckpoint(OUT_MODELS_FOLDER + checkpoint_filename, 'val_loss', save_best_only=True)
}

model, history = execute_pipeline(conv_base, train_dataset, val_dataset, hyperparameters)

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_9 (InputLayer)        [(None, 180, 180, 3)]     0         
                                                                 
 sequential_2 (Sequential)   (None, 180, 180, 3)       0         
                                                                 
 inception_resnet_v2 (Funct  (None, 4, 4, 1536)        54336736  
 ional)                                                          
                                                                 
 flatten_3 (Flatten)         (None, 24576)             0         
                                                                 
 dense_7 (Dense)             (None, 1024)              25166848  
                                                                 
 dropout_4 (Dropout)         (None, 1024)              0         
                                                                 
 dense_8 (Dense)             (None, 5)                 5125      
                                                                 
=================================================================
Total params: 79508709 (303.30 MB)
Trainable params: 29302309 (111.78 MB)
Non-trainable params: 50206400 (191.52 MB)
_________________________________________________________________
Epoch 1/50
379/379 [==============================] - 56s 127ms/step - loss: 0.2673 - accuracy: 0.9024 - f1_score: 0.9024 - precision_5: 0.9321 - recall_5: 0.8773 - val_loss: 0.0934 - val_accuracy: 0.9710 - val_f1_score: 0.9715 - val_precision_5: 0.9782 - val_recall_5: 0.9644
Epoch 2/50
379/379 [==============================] - 47s 121ms/step - loss: 0.1610 - accuracy: 0.9405 - f1_score: 0.9406 - precision_5: 0.9506 - recall_5: 0.9309 - val_loss: 0.0916 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_5: 0.9753 - val_recall_5: 0.9659
Epoch 3/50
379/379 [==============================] - 47s 121ms/step - loss: 0.1371 - accuracy: 0.9491 - f1_score: 0.9492 - precision_5: 0.9563 - recall_5: 0.9428 - val_loss: 0.0915 - val_accuracy: 0.9688 - val_f1_score: 0.9693 - val_precision_5: 0.9745 - val_recall_5: 0.9651
Epoch 4/50
379/379 [==============================] - 46s 120ms/step - loss: 0.1247 - accuracy: 0.9537 - f1_score: 0.9538 - precision_5: 0.9593 - recall_5: 0.9480 - val_loss: 0.0802 - val_accuracy: 0.9725 - val_f1_score: 0.9729 - val_precision_5: 0.9754 - val_recall_5: 0.9703
Epoch 5/50
379/379 [==============================] - 43s 113ms/step - loss: 0.1106 - accuracy: 0.9590 - f1_score: 0.9591 - precision_5: 0.9637 - recall_5: 0.9548 - val_loss: 0.0843 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_5: 0.9768 - val_recall_5: 0.9696
Epoch 6/50
379/379 [==============================] - 43s 113ms/step - loss: 0.1022 - accuracy: 0.9642 - f1_score: 0.9643 - precision_5: 0.9681 - recall_5: 0.9599 - val_loss: 0.0858 - val_accuracy: 0.9651 - val_f1_score: 0.9656 - val_precision_5: 0.9679 - val_recall_5: 0.9629
Epoch 7/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0996 - accuracy: 0.9618 - f1_score: 0.9619 - precision_5: 0.9664 - recall_5: 0.9582 - val_loss: 0.0816 - val_accuracy: 0.9688 - val_f1_score: 0.9694 - val_precision_5: 0.9732 - val_recall_5: 0.9688
Epoch 8/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0875 - accuracy: 0.9692 - f1_score: 0.9692 - precision_5: 0.9723 - recall_5: 0.9655 - val_loss: 0.0809 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_5: 0.9760 - val_recall_5: 0.9673
Epoch 9/50
379/379 [==============================] - 46s 120ms/step - loss: 0.0855 - accuracy: 0.9703 - f1_score: 0.9704 - precision_5: 0.9732 - recall_5: 0.9678 - val_loss: 0.0794 - val_accuracy: 0.9718 - val_f1_score: 0.9722 - val_precision_5: 0.9761 - val_recall_5: 0.9688
Epoch 10/50
379/379 [==============================] - 47s 122ms/step - loss: 0.0843 - accuracy: 0.9671 - f1_score: 0.9672 - precision_5: 0.9700 - recall_5: 0.9648 - val_loss: 0.0778 - val_accuracy: 0.9718 - val_f1_score: 0.9723 - val_precision_5: 0.9738 - val_recall_5: 0.9673
Epoch 11/50
379/379 [==============================] - 46s 121ms/step - loss: 0.0707 - accuracy: 0.9734 - f1_score: 0.9735 - precision_5: 0.9759 - recall_5: 0.9718 - val_loss: 0.0770 - val_accuracy: 0.9710 - val_f1_score: 0.9715 - val_precision_5: 0.9731 - val_recall_5: 0.9681
Epoch 12/50
379/379 [==============================] - 43s 113ms/step - loss: 0.0690 - accuracy: 0.9742 - f1_score: 0.9742 - precision_5: 0.9767 - recall_5: 0.9720 - val_loss: 0.0930 - val_accuracy: 0.9666 - val_f1_score: 0.9670 - val_precision_5: 0.9680 - val_recall_5: 0.9651
Epoch 13/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0660 - accuracy: 0.9757 - f1_score: 0.9757 - precision_5: 0.9779 - recall_5: 0.9741 - val_loss: 0.0899 - val_accuracy: 0.9718 - val_f1_score: 0.9721 - val_precision_5: 0.9725 - val_recall_5: 0.9710
Epoch 14/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0624 - accuracy: 0.9780 - f1_score: 0.9780 - precision_5: 0.9804 - recall_5: 0.9759 - val_loss: 0.0804 - val_accuracy: 0.9762 - val_f1_score: 0.9766 - val_precision_5: 0.9784 - val_recall_5: 0.9755
Epoch 15/50
379/379 [==============================] - 43s 113ms/step - loss: 0.0583 - accuracy: 0.9787 - f1_score: 0.9788 - precision_5: 0.9808 - recall_5: 0.9768 - val_loss: 0.0921 - val_accuracy: 0.9681 - val_f1_score: 0.9685 - val_precision_5: 0.9695 - val_recall_5: 0.9681
Epoch 16/50
379/379 [==============================] - 43s 112ms/step - loss: 0.0606 - accuracy: 0.9777 - f1_score: 0.9778 - precision_5: 0.9803 - recall_5: 0.9758 - val_loss: 0.0809 - val_accuracy: 0.9703 - val_f1_score: 0.9707 - val_precision_5: 0.9710 - val_recall_5: 0.9703

4. Discussion and results comparison.

Along this notebook step-by-step explanations were provided of the followed steps during the project. To conclude we will discuss the most relevant steps and compare the performance of the best model obtained for each approach.

Multiple different configurations have been tested. First of all, we have trained an arbitrary baseline model as a starting point to check whether CNNs are suitable for the problem to handle, having only the 2 blocks of convolution and max-pooling layers, plus the final classifier with denses layers.

As expected, CNN were a good option for image classification since they initially provided a quite good accuracy without receiving a task tuning.

Our first approximation consisted in creating a CNN architecture from scratch, so we did several experiments to find an architecture suitable for our problem following tuning process and tracking the validation accuracy. We concluded that the best architecture was {'input_units': 64, 'kernel_size': 3, 'n_layers': 3, 'n_conv': 1, 'n_connections': 3, 'n_nodes': 128} and we proceeded to train this baseline model. However, it started to overfit very soon. Concretely in epoch 5, point from which the validation loss started to increase, while the training loss kept decreasing.

The overall metrics were quite acceptable, with an accuracy of 83.60% in validation. Observing that the training loss and training accuracy was still improving, we tried to regularize the network in order to let the validation performance to be improved as well. First of all, we tried using batch normalization layers after the convolutional layers to make the training faster and more stable. However, it did not seem to reduce overfitting and even affected the loss curve making the training much more inestable. On the other hand, using dropout and data normalization seemed to affect positively the training, specially the former one which achieved to delay the overfitting emergence to the epoch 7 and obtaining a validation accuracy of ~0.82, slighly lower than the baseline model.

Regarding weight regularization, L1 regularization makes the model completely unable to learn the patterns from the dataset, returning a validacion_accuracy of 0.2 (random classification since there are 5 classes). L1 regularization is generally considered to generate models that are simple and interpretable but not capable of learning complex patterns, which highlights the necesity of learning complex patterns in our dataset. The same occurs when combined with L2.

However, applying L2 regularization we achieved to stabilize the training delaying the overfitting emergence until epoch 12 and obtaining a validation accuracy of 0.8159. We tryied then combining some of the techniques which seemed to deal with the overfitting problem, such as Dropout with Data augmentation in which the model updated the weights too drastically causing a inestability in the training and which make us aware of the importance of the learning rate parameter.

Reducing it to 0.0005 allowed to stabilize the training and obtain a ~0.86 validation accuracy avoiding early overfitting problems as well. Having as a goal to reach the 90% in validation accuracy, we decided to repeat the previous process adding a learning rate scheduler to progressively reduce the learning rate value (Learning rate decay) and then test adding L2 regularization which provided good results in the previous experiments.

The combination of dropout with data augmentation and learning rate scheduler provided the best results for the scratch model with a validation accuracy around 0.88. However adding L2 regularization, though it did not affect to the training stability, reduced considerably the classification performance.

Our second approximation consisted in repeating the previous experiments but with a different baseline architecture (random) just to detect whether the architecture tuning process is actually helpful for DL trainings. We obtained similar problems to the first architecture and at the end we were not able to surpass the previous performance results having in general much lower metrics and highlighting the importance of the initial architecture tuning process for finding the architecture most suitable for our problem.

Our third approximation consisted in testing with more complex architectures (not pretrained), concretely VGG16, VGG19, Inception, InceptionResNetV2, Xception and Residual. The training using VGG16 and VGG19 was not succesfull and we attached this problem to the huge complexity of these architectures with respect to our small dataset. They are composed by a large number of parameters and they were originally trained on the ImageNet dataset, which contains millions of images across thousands of categories. A small dataset does not provide enough diverse examples for the model to learn robust features and generalize well to unseen data.

For the rest of the models in general the results were quite similar to the ones obtained when training the model from scratch (with a validation accuracy near ~0.8). However, Xception model achieved to reach the 0.9 of validation accuracy. Note that for Inception and Residual networks we had to repeat the training because of a too high learning rate, similar as a problem occured when training the architecture from scratch.

Our fourth approximation consisted in creating a wide model using Depthwise separable convolutions. As always, we first trained the baseline model and then we applyied regularization techniques. The baseline networks started to overfit in epoch 11 so we deciced to try the model adding dropout and data augmentation since they previously demonstrated more appropiate evolution of the curves. Both techniques improved considerable the results (from 0.8 to 0.9 of validation accuracy) and adjusted training and validation curves almost perfectly.

Our fifth approximation consisted in applying transfer learning for three pre-trained models: VGG16, VGG19 and an additional experiment using the best convolutional model trained from scratch in the previous section. We followed two different transfer learning paradigms trying to maximize the performance of those models: feature extraction and fine-tuning.

The VGG16 baseline model was a really good starting point since it provides very good results (validation accuracy: 0.9117) overcoming our previous model defined from scratch. However the training is stopped soon because the network starts to overfit in epoch 3. Thus we tryied to reduce the overfitting problem using dropout and data augmentation. Testing this selected regularized network, the overfitting problem had been highly reduced since the validation loss kept almost horizontal. Nevertheless, we could observe some peaks derived from a high learning rate value, so we used again the learning rate scheduler to reduce this problem and train again reaching a validation accuracy of 0.9198. The rest of the models provided similar results being InceptionResNetV2 the one which achieves the best validation results (~0.97).

Then, we tested the fine-tuning approach carrying out experiments training 1 and 2 convolutional blocks. Using the baseline model without regularization we could see good validation results, reaching validation accuracies of ~0.94. However when we applyied regularization we achieved to improved the performance even more. In general fine-tuning 2 layers seemed to provide better validation results.

Regarding the experiment with the best model trained from scratch in the previous section using feature extraction. It was interesting to see that the loss curve remained constant across epochs due to the network had already learned features specific to the dataset during the initial training. However, it is always a good practice to experiment with different approaches and evaluate their performance empirically. Sometimes, even if the initial features learned during pre-training overlap with the dataset, fine-tuning certain layers or adapting the network's architecture can still lead to improvements in performance.

To end with this section we will analyze test results of the best models obtained from validation results. The following plot sets a comparison among them.

def metric_comparison_plot(model_names, metric_name, val, test):
    num_models = len(val)

    # Plot the results
    plt.figure(figsize=(14, 6))
    bar_width = 0.35
    index = np.arange(num_models)

    plt.bar(index, val, bar_width, label='Validation '+metric_name)
    plt.bar(index + bar_width, test, bar_width, label='Test '+metric_name)

    plt.xlabel('Models')
    plt.ylabel(metric_name)
    plt.title('Animal classification results')
    plt.xticks(index + bar_width / 2, [label.replace('_', '_\n') for label in model_names])
    plt.legend()

    plt.tight_layout()
    plt.show()

models = [f for f in os.listdir(OUT_MODELS_FOLDER) if f.endswith('.keras')]

# Reserve memory
model_names, val_accuracies, test_accuracies, val_f1s, test_f1s, val_precisions, test_precisions, val_recalls, test_recalls = [], [], [], [], [], [], [], [], []

# Get results
for model in models:
    test_model = keras.models.load_model(OUT_MODELS_FOLDER+"/"+model)
    val_loss, val_acc, val_f1, val_precision, val_recall = test_model.evaluate(val_dataset)
    test_loss, test_acc, test_f1, test_precision, test_recall = test_model.evaluate(test_dataset)

    model_names.append(model)
    
    val_accuracies.append(val_acc)
    val_f1s.append(val_f1[0])
    val_precisions.append(val_precision)
    val_recalls.append(val_recall)

    test_accuracies.append(test_acc)
    test_f1s.append(test_f1[0])
    test_precisions.append(test_precision)
    test_recalls.append(test_recall)

# Create plots
metric_comparison_plot(model_names, "Accuracy",val_accuracies,test_accuracies)
metric_comparison_plot(model_names, "F1",val_f1s,test_f1s)
metric_comparison_plot(model_names, "Precision",val_precisions,test_precisions)
metric_comparison_plot(model_names, "Recall",val_recalls,test_recalls)

43/43 [==============================] - 2s 44ms/step - loss: 0.4815 - accuracy: 0.8107 - f1_score: 0.8114 - precision_1: 0.8434 - recall_1: 0.7795
47/47 [==============================] - 2s 43ms/step - loss: 0.3648 - accuracy: 0.8597 - f1_score: 0.8598 - precision_1: 0.9004 - recall_1: 0.8210
43/43 [==============================] - 2s 48ms/step - loss: 0.4626 - accuracy: 0.8233 - f1_score: 0.8234 - precision_1: 0.8586 - recall_1: 0.7892
47/47 [==============================] - 2s 43ms/step - loss: 0.3832 - accuracy: 0.8524 - f1_score: 0.8509 - precision_1: 0.8857 - recall_1: 0.8176
43/43 [==============================] - 2s 48ms/step - loss: 0.3680 - accuracy: 0.8738 - f1_score: 0.8758 - precision_3: 0.8849 - recall_3: 0.8619
47/47 [==============================] - 2s 43ms/step - loss: 0.3141 - accuracy: 0.8978 - f1_score: 0.8984 - precision_3: 0.9087 - recall_3: 0.8838
43/43 [==============================] - 2s 48ms/step - loss: 0.2467 - accuracy: 0.9079 - f1_score: 0.9084 - precision_3: 0.9260 - recall_3: 0.8827
47/47 [==============================] - 2s 44ms/step - loss: 0.1830 - accuracy: 0.9332 - f1_score: 0.9331 - precision_3: 0.9484 - recall_3: 0.9205
43/43 [==============================] - 11s 147ms/step - loss: 0.0885 - accuracy: 0.9710 - f1_score: 0.9716 - precision_4: 0.9738 - recall_4: 0.9666
47/47 [==============================] - 7s 148ms/step - loss: 0.0991 - accuracy: 0.9673 - f1_score: 0.9671 - precision_4: 0.9711 - recall_4: 0.9646
43/43 [==============================] - 9s 88ms/step - loss: 0.0770 - accuracy: 0.9710 - f1_score: 0.9715 - precision_5: 0.9731 - recall_5: 0.9681
47/47 [==============================] - 5s 91ms/step - loss: 0.0939 - accuracy: 0.9693 - f1_score: 0.9691 - precision_5: 0.9712 - recall_5: 0.9679
43/43 [==============================] - 4s 58ms/step - loss: 0.0957 - accuracy: 0.9651 - f1_score: 0.9657 - precision_6: 0.9658 - recall_6: 0.9644
47/47 [==============================] - 3s 60ms/step - loss: 0.0784 - accuracy: 0.9739 - f1_score: 0.9739 - precision_6: 0.9771 - recall_6: 0.9706
43/43 [==============================] - 4s 57ms/step - loss: 0.1993 - accuracy: 0.9235 - f1_score: 0.9242 - precision: 0.9393 - recall: 0.9079
47/47 [==============================] - 3s 58ms/step - loss: 0.1715 - accuracy: 0.9372 - f1_score: 0.9372 - precision: 0.9513 - recall: 0.9259

# Create a table
table_data = [["Model", "Validation Accuracy", "Test Accuracy"]]
for i in range(len(model_names)):
    row = [models[i],val_accuracies[i],test_accuracies[i]]
    table_data.append(row)

# Print the table
print(tabulate(table_data, headers="firstrow", tablefmt="grid"))

+----------------------------------------------------+-----------------------+-----------------+
| Model                                              |   Validation Accuracy |   Test Accuracy |
+====================================================+=======================+=================+
| cnn2_reg_lr0.0005_dropout_aug_lrScheduler.keras    |              0.81069  |        0.859719 |
+----------------------------------------------------+-----------------------+-----------------+
| cnn_baseline_lr0.001.keras                         |              0.823311 |        0.852371 |
+----------------------------------------------------+-----------------------+-----------------+
| cnn_reg_lr0.0005_dropout_aug_lrScheduler.keras     |              0.873794 |        0.897796 |
+----------------------------------------------------+-----------------------+-----------------+
| cnn_wide_lr0.001_dropout_da.keras                  |              0.907944 |        0.9332   |
+----------------------------------------------------+-----------------------+-----------------+
| InceptionResNetV2_feature_extraction_dropout.keras |              0.971047 |        0.967268 |
+----------------------------------------------------+-----------------------+-----------------+
| InceptionResNetV2_finetuning2_dropout_da.keras     |              0.971047 |        0.969272 |
+----------------------------------------------------+-----------------------+-----------------+
| Xception_finetuning_dropout_da.keras               |              0.965108 |        0.973948 |
+----------------------------------------------------+-----------------------+-----------------+
| xception_lr0.001.keras                             |              0.923534 |        0.937208 |
+----------------------------------------------------+-----------------------+-----------------+

Below is the relationship between the previous models and the different experiments carried out to facilitate their interpretation:

• cnn2_reg_lr0.0005_dropout_aug_lrScheduler.keras: One random architecture from scratch trained after applying regularization
• cnn_baseline_lr0.001.keras: The best architecture from scratch obtained after architecture tuning
• cnn_reg_lr0.0005_dropout_aug_lrScheduler.keras: The best architecture from scratch obtained after architecture tuning after applying regularization
• cnn_wide_lr0.001_dropout_da.keras: The best wide model using Depthwise separable convolutions trained from scratch after applying regularization
• xception_lr0.001.keras: The best complex model trained from scratch
• InceptionResNetV2_feature_extraction_dropout.keras: The best complex model after applying feature extraction and regularization
• Xception_finetuning_dropout_da.keras: The best complex model after applying fine-tuning (1 convolutional block) and regularization
• InceptionResNetV2_finetuning2_dropout_da.keras: The best complex model after applying fine-tuning (2 convolutional blocks) and regularization

Here are some observations and insights from the previous results:

• Model Performance: The models show varying performance in terms of both validation and test accuracies. This situation could be result from a data splitting discrepancy since the validation set might be significantly harder or different from the test set. If the validation set is not representative of the test set, it can lead to differences in accuracy metrics. This can happen because the training data splitting was done randomly, and the validation set by chance contains more difficult examples.
• Model Complexity: More complex models like Xception seem to perform better, both in terms of validation and test accuracies, than the trained from scratch ones.
• Depthwise separable convolutions are really powerful mechanisms since, with significantly fewer parameters and fewer computations compared to regular convolution, obtain even higher performance results.
• Regularization: Models with regularization techniques such as dropout seem to perform better.
• Transfer learning: Since the baseline models trained from scratch gave worse results than using a pre-trained (regularized or not) model, we concluded that the transfer learning technique was successful for both, feature extraction and fine-tuning, and the features learned from the models during pre-training were succesfully leveraged in our classification task.
• Fine-tuning vs. Feature Extraction: Transfer learning approaches provided the best results along this project, reaching an impressive performance of 0.97 test accuracy (e.g Xception_finetuning_dropout_da.keras). Fine-tuning pre-trained models reach to improve a bit the results in test accuracy compared to feature extraction but that difference is not really high, at least in this problem. Thus we can say that fine-tuning is better, but not too much.
• Learning Rate: Learning rate values (e.g., lr0.001, lr0.0005) affect the model's performance. However, the impact varies based on the architecture and training strategy. It is important to consider this parameter value carefully.